What counts as good evidence?

Since this is a search for evidence that will help answer ultimate questions, we need to consider relevant and valid evidence. A good start is the following critical thinking guidelines from a book titled How to Think About Weird Things:

• Just because something is logically possible doesn't mean that it's real.
• Just because a claim hasn't been conclusively refuted doesn't mean that it's true.
• Just because a claim hasn't been conclusively proven doesn't mean that it's false.
• Just because you can't explain something doesn't mean that it's supernatural.
• Just because something is physically possible doesn't mean that it's real.
• Just because something seems (feels, appears) real doesn't mean that it is.
• Just because you believe something to be true doesn't mean that it is.
• Just because a group of people believe that something is true doesn't mean that it is.
• There is an external reality that is independent of our representations of it.
• There is good reason to doubt a proposition if it conflicts with other propositions we have good reason to believe.
• The more background information a proposition conflicts with, the more reason there is to doubt it.
• When there is good reason to doubt a proposition, we should proportion our belief to the evidence.
• There is good reason to doubt a proposition if it conflicts with expert opinion.
• Just because someone is an expert in one field doesn't mean that he or she is an expert in another.
• If we have no reason to doubt what's disclosed to us through perception, introspection, memory, or reason, then we're justified in believing it.
• When evaluating a claim, look for disconfirming as well as confirming evidence.
• When evaluating a claim, look at all the relevant evidence, not just the psychologically available evidence.
• A hypothesis is scientific only if it is testable, that is, only if it predicts something other than what it was introduced to explain.
• Other things being equal, the best hypothesis is the one that is the most fruitful, that is makes the most novel predictions.
• Other things being equal, the best hypothesis is the one that has the greatest scope, that is, that explains and predicts the most diverse phenomena.
• Other things being equal, the best hypothesis is the simplest one, that is, the one that makes the fewest assumptions.
• Other things being equal, the best hypothesis is the one that is the most conservative, that is, the one that fits best with established beliefs.
• We should accept an extraordinary hypothesis only if no ordinary one will do.
• Personal experience alone generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• Case studies alone generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• When claims of a treatment's effectiveness are based solely on case studies or personal experience, you generally cannot know that the treatment is effective.
• Scientific evidence gained through controlled experiments, unlike personal experience and case studies, generally can establish the effectiveness of a treatment beyond a reasonable doubt.
• Single medical studies generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• When the results of relevant studies conflict, you cannot know that the treatment in question is effective.
• New study results that conflict with well-established findings cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• Test-tube studies alone generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• Animal studies alone generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• Observational studies alone generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.
• Clinical trials limited by lack of a control group, faulty comparisons, or small numbers generally cannot establish the effectiveness of a treatment beyond a reasonable doubt.