define random hypothesis

Random Walk Hypothesis

• First Online: 01 September 2021

Cite this chapter

• Moinak Maiti 2  

1365 Accesses

1 Citations

This chapter covers history, definition, assumptions, and implications of the Random walk hypothesis. The basic idea is that stock prices take a random and unpredictable path. Discussion includes why the random walk hypothesis is still relevant in finance in spite of several criticisms? Detailed discussion made on random walk hypothesis and market efficiency. Fama’s joint hypothesis problem and its implication is covered in detail. In addition to it martingales and its features are conversed in detail. Illustrations are shown for different random walk models using R-Programming namely Random Walk with Fixed Moves and Random Walk with Random Moves. Critical issues related to the various Random Walk models in practice are discussed in detail. At the end testing of the various Random Walk models and Martingales using EViews are demonstrated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

https://archive.org/details/calculdeschances00regn/page/50/mode/2up , accessed on 31/03/2021.

Bachelier, L. (1900). Théorie de la spéculation. In Annales scientifiques de l’École normale supérieure (Vol. 17, pp. 21–86).

Eugene, F. (1991). Efficient capital markets: II. Journal of Finance, 46 (5), 1575–1617.

Article   Google Scholar  

Fama, E. F. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25 (2), 383–417.

Kim, J. H. (2006). Wild bootstrapping variance ratio tests. Economics Letters, 92 (1), 38–43.

Lo, A. W., & MacKinlay, A. C. (1988). Stock market prices do not follow random walks: Evidence from a simple specification test. The Review of Financial Studies, 1 (1), 41–66.

Maiti, M. (2020). A critical review on evolution of risk factors and factor models. Journal of Economic Surveys, 34 (1), 175–184.

Download references

Author information

Authors and affiliations.

Department of Finance, National Research University Higher School of Economics, St. Petersburg, Russia

Moinak Maiti

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Moinak Maiti .

2.1.1 Multiple Choice Questions

EMH does not assume

Security prices follow a random walk

Security prices follow a martingale

Security prices do not follow a random walk

Which of the following statement(s) about martingales is/are false?

Martingale and random walk are the same

Martingale is not a stochastic process

Martingales are the sequences of random variables

All of the above

Which of the following version of EMH does not exists?

Semi Weak form

Semi Strong form

Strong form

Dart Throwing Investment Contest initiated in which year?

Which of the following test is not associated with testing random walk?

Variance ratio test

Wald (Chi-Square) joint tests

Chow-Denning maximum |z| joint tests

BDS Independence test

The random walk hypothesis is mostly related to the

Weak form of EMH

Semi Weak form of EMH

Semi Strong form of EMH

Strong form of EMH

An investor notices a particular trend in a security price movements. This is a violation of the

Which of the following test is used for testing random walk?

Phillips-Perron test

BDS independence test

Which among the following test is used for autocorrelation analysis?

Kendall’s tau

Ljung-Box test

Pearson correlation

Shapiro–Wilk test

According to Eugene Fama test of the market efficiency is difficult due to

Joint hypothesis problem

Forward hypothesis problem

Backward hypothesis problem

None of the above

2.1.2 Fill in the Blanks

Market efficiency should be tested jointly with the __________ & __________.

Test of market efficiency seems to be impossible due to __________ problem.

____________ test is used for autocorrelation analysis.

Dart Throwing Investment Contest initiated by _________ in the year ______.

EMH stands for ____________________.

\(E({\mathrm{Price}}_{t+1}- {\mathrm{Price}}_{t}|{\Phi}_{t})=0\) , represents a ______________.

In the following random walk model: \({Price}_{t}= {Price}_{t-1}\pm {\alpha }_{t}\) , \({\alpha }_{t}\) represents ________________.

The arrival of relevant new information is a ___________ process.

Presence of the stock market anomalies evidence against the _________.

Martingale follows a ___________ process.

2.1.3 Long Answer Questions

Define Random walk hypothesis with suitable examples?

Define Efficient Market Hypothesis with suitable examples?

What are the three versions of EMH and its implications? Explain it in brief.

Develop a random walk model with random drifts using Poisson’s ratio and execute it with R programming to obtain the estimates?

Develop a random walk model with random drifts as the log returns and execute it with R programming to obtain the estimates?

Revisit Fig.  2.2 and replot it considering unequal probability of getting a head or tail?

Revisit Fig.  2.3 and replot it with the mean value equals to 0.02? Check the difference and comments on its implications.

Comment on the statement “Are markets really efficient during COVID-19 pandemic” with suitable examples.

Using different R operators define a martingale and execute it in R console?

What is your understanding on Joint Hypothesis Problem? Explain it in brief.

What are public and private information?

Discuss EMH with respect to the technical analysis and fundamental analysis?

A researcher wants to conduct a study to understand whether the security prices follow a random walk during the financial crisis 2008 and COVID-19 on the US and UK financial markets? But the researcher does not know from where to begin. So, help him/her to begin with the analysis.

A central banker want to examine whether the “USD_EUR” daily exchange rates for the past five years follow a martingale? Put yourself in the place of central banker and finish the task?

Define: what is a non-random walk?

2.1.4 Real-World Tasks

An instructor ask his/her student to test for the market efficiency during the COVID-19 first and second phase impact on the European stock markets? Assume yourself as the student and perform the task. Then prepare a detailed report of the analysis to be submitted to the instructor.

A senior researcher want to analyse whether the top seven cryptocurrencies returns follow a random walk during the noble Coronavirus pandemic. Assume yourself as the researcher: perform the mentioned task in details and develop the analysis report.

An investment analyst want to examine whether the world indices before, during, and after the subprime crisis follow a martingale. Help him/her to perform the said analysis and prepare the report.

A student need to test the weak form of EMH for the major three currency pairs daily exchange rate for the last three years as his/her project dissertation. Help the student to complete his/her project dissertation successfully and satisfactorily.

A senior manager of the Reserve bank of India (RBI) asked an intern working under him/her to conduct semi-strong form of EMH tests for all the technological securities traded in the BSE during the first phase of COVID-19 pandemic. Help the intern in analysing and developing the final report for timely submission to the senior RBI manager.

An individual investor is evaluating his/her investing option as the Herzfeld Caribbean Basin Fund (CUBA), a closed ended fund during early 2021. Help the investors to examine the CUBA fund with respect to the EMH and help him/her to make investment decision.

Conduct a simple empirical research to show that the security prices do react to the news announcements.

Conduct a simple empirical research that evidences against the EMH?

2.1.5 Case Studies

A researcher want to conduct a research to test whether stock market reacts to the government policy announcement. During the COVID-19 pandemic Indian government has taken several important policies to tackle the pandemic situation. In that aspect the researcher decided to test the Indian stock market efficiency with respect to the impact of Indian government important policies announcement between the period of January 2021 to September 2021.

Consider yourself in place of the researcher conduct the research and prepare a detail report based on your analysis.

Below figures shows the Variance ratio test estimates and descriptive statistics for the GBP_USD currency pair daily exchange rates between the period 01/01/2021 to 31/03/2021. Based on these data comments on the market efficiency and distribution of the data. Then also comment on the relationship between the volatility and market efficiency if any.

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Maiti, M. (2021). Random Walk Hypothesis. In: Applied Financial Econometrics. Palgrave Macmillan, Singapore. https://doi.org/10.1007/978-981-16-4063-6_2

Download citation

DOI : https://doi.org/10.1007/978-981-16-4063-6_2

Published : 01 September 2021

Publisher Name : Palgrave Macmillan, Singapore

Print ISBN : 978-981-16-4062-9

Online ISBN : 978-981-16-4063-6

eBook Packages : Economics and Finance Economics and Finance (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Recently Active
• Top Discussions
• Best Content

By Industry

• Investment Banking
• Private Equity
• Hedge Funds
• Real Estate
• Venture Capital
• Asset Management
• Equity Research
• Investing, Markets Forum
• Business School
• Fashion Advice
• Technical Skills
• Trading & Investing Guides

Random Walk Theory

It is a financial theory that states that the stock prices in a stock market are independent of their historical trends.

Christy currently works as a senior associate for EdR Trust, a publicly traded multi-family  REIT . Prior to joining EdR Trust, Christy works for CBRE in investment property sales. Before completing her  MBA  and breaking into finance, Christy founded and education startup in which she actively pursued for seven years and works as an internal auditor for the U.S. Department of State and CIA.

Christy has a Bachelor of Arts from the University of Maryland and a Master of Business Administrations from the University of London.

Currently an investment analyst focused on the  TMT  sector at 1818 Partners (a New York Based Hedge Fund), Sid previously worked in private equity at BV Investment Partners and BBH Capital Partners and prior to that in investment banking at UBS.

Sid holds a  BS  from The Tepper School of Business at Carnegie Mellon.

What Is Random Walk Theory?

• How Does The Random Walk Theory Work?

Pros And Criticisms Of Random Walk Theory

• Random Walk Theory In The Stock Market

Random walk theory or random walk hypothesis is a financial theory that states that the stock prices in a stock market are independent of their historical trends. This means that the prices of these securities follow an irregular trend.

The theory further states that the future prevailing prices of a stock cannot be predicted accurately even after deploying its historical prices. 

It all started with a mathematician,  Jules Regnault,  who turned into a stockbroker and wrote a book named “Calcul des Chances et Philosophie de la Bourse” or “The Study of Chance and the Philosophy of Exchange.” 

He pioneered the application of mathematics to the stock market for stock market analysis. Another French mathematician, Louis Bachelier, published his paper, “Théorie de la Spéculation,” or the “Theory of Speculation.” 

This theory has been named after the book  “A Random Walk Down Wall Street,”  authored by American economist   Burton Malkiel . 

His theory about the stock market that the stock prices follow an unpredictable random path was mainly criticized by many experts in America alone and worldwide too. 

This theory works over the following two broad assumptions:

• The price of the securities on the stock market follows a random trend
• There is no dependency between any two securities being traded in the stock market

We will discuss in detail the criticism of the foundation built by these assumptions later in the coming sections.

An interesting point to note here is that Malkiel himself mentioned the term “ efficient market ” while delineating his theory that plays a crucial role.

He says that stock prices cannot be predicted through any analysis in a stock market as they follow a random path. 

Let’s delve into this a tad bit. 

An efficient market, by definition, is a market where all the information required for trading in the stock market is freely available; that is, the market is transparent, and all the players have equal access to general information about the stocks being traded in the market.

The definition here states that an efficient market allows a trader to analyze all the necessary information about the stock they plan to buy after proper strategic planning through the information available. 

Let us hold onto that thought and move to the next section. It’s going to be an interesting yet mind-boggling conversation! 

how does the Random Walk Theory work?

Although criticized by many stock analysts, Random Walk Theory has proven itself in the past. The most recent one was the outbreak of COVID-22, which was completely unanticipated and out of control. 

This theory states that if the stock prices are random, then the prediction of the future standing of the stock cannot be made. 

Fund managers are the people whom a company hires to perform predictive analyses of stock prices based on their recent historical records. 

Let us say that the managers’ forecasts work out, and the company earns a considerable profit. For this, there can be two explanations at large. 

First, the forecasting and analysis done by the experts were accurate enough to let the company earn that profit.

Second, it could be possible that the markets favored the company during that period, and the profits were bagged due to coincidence and luck. 

As for the first statement made in that respect, the question arises that if the predictions are positive, the company is more likely to get huge returns, then why is it so that even after rigorous research techniques and models, they end up earning just the face value or even lower?  

Furthermore, for the second statement, it could be that the market showed trends in favor of the company leading them to earn alpha returns. 

The random walk theory simply says that these profits are earned by nothing but chance, as the stock prices cannot be determined for the future. 

We are now coming back to our previous conversation on efficient markets . As we mentioned, an efficient market provides any information required for analyzing the stocks of a company. 

This implies that the future price of stocks can be predicted with the help of their historical movements and trends, given that the market is efficient. 

If this is true, why do some expert analysts bear losses and others do not?

This would only give one the impression that either loss or gain, it’s a matter of luck. 

However, experts disagree with this theory. In a world where a computer achieves the impossible, algorithms come in handy while trading. 

A strong algorithm will bear more fruits, and a weak one will bear more losses. 

Let’s take a real-time example to illustrate Random Walk Theory. 

We all recollect when COVID-19 hit the world economy , and the stock markets thrashed worldwide. This was a sudden happening and a factor that was out of everyone’s control.

Even the investors who have practicing investment for a long time and who knew their strategies and their way into and out of the market could not do anything but bear huge losses. 

This is one of those instances where no pre-planned strategy or a strong algorithm could improve one’s market standing in such adversities.

Since every theory has advantages and disadvantages, Random Walk Theory lists specific pros and cons to keep in mind.

A few of the pros and cons are:

• Markets are not entirely efficient. This means the information required to invest in that stock is not fully transparent.
• Many insiders acquire information before the general public investors, giving them an edge over others. To overcome this catch, one should invest directly in ETFs (Exchange Traded Funds) to earn decent returns. ETFs are funds that hold multiple types of securities as their underlying asset . An ETF is an excellent option for a risk-averse person as it allows the option of portfolio diversification . You can read more about ETFs  here . 
• Moreover, historical trends depict that a stock price can easily fluctuate by even significantly irrelevant news about a company. Since sentiments can’t be predicted, the stock movements are unpredictable and erratic.
• Random Walk Theory is gaining popularity among passive investors as the fund managers fail to outperform the index and increase the investors’ increasing belief in randomness. 
• Random Walk Theory argues that it is not entirely impossible to outperform the market. However, one can efficiently earn a decent amount of profits with careful analysis and possible future happenings related to that particular stock.
• They further state that this theory is baseless and stocks instead follow a trend that could be deduced from historical data, combined with certain future possible factors that might affect the stock movement. 
• There may be an infinite number of factors influencing the stock movement that might make the task of detecting the pattern pretty cumbersome. But this, they argue, doesn’t mean that if a pattern is not quickly visible, it doesn’t exist. 

Random walk theory in the stock market

It’s time to resume the conversation that we began in the first section. 

We’ve only talked about how this theory states that stock movements are unpredictable and that one cannot outperform the market index by any financial analysis. 

When we want to earn money quickly, we invest in stocks. Then, we wait for them to grow over time and pull back the profits. The rebuttal here is that if earning through investing in stocks is that easy, why do many investors end up losing their money? 

If Random Walk Theory doesn’t stand any chance in the practicality of stock markets, why are some investors able to bag profits from the same stock while other investors bear losses?

This is all about the timing and future growth probability of that particular company. The best profits are earned when the market is unstable and erratic. 

The investor needs to plan his standing in the market in such a strategic way that the market favors them. Moreover, RWT doesn’t consider the concept of insider information or an information edge which, in turn, refutes the fact that stocks are random. 

In essence, today’s stock price is independent of yesterday’s stock price, an aggregate result of information available at that time. Due to the failure of many fund managers to generate sufficient profits over stocks, there has been an increase in the number of investors in index funds .

Index funds are a kind of mutual fund or ETF that tracks the market value of a specified basket of underlying assets.

Click here to read more about  Index Funds . All in all, trading is assumed between informed buyers and sellers with completely different strategies. Thus, ultimately the market follows a random path.

So, we have dwelled on the intricacies of this stock theory thus far. What do you think was a reasonable argument with respect to or against this theory? 

All these years, the debate over this theory has never had a hiatus among its fanatics and critics. Each individual depends mainly on how much investment has been put into the stock market. 

The only way to deal with this problem is to act according to what is best for you as an investor. For example, if you believe that stock movement is random, you should invest your money into ETFs, which reflect the whole market return in just one portfolio. 

ETFs, as told earlier, are more diverse as they constitute a mix of assets that yield better returns with minimal risk when blended into one single portfolio. Thus, this option could be used by risk-averse investors.

Now, if you are the other half, you think that stocks are not random but rather form a pattern through which one can predict future prices; you should perform financial analysis and generate return predictions for future periods for your investment.

After all, investing some of your wealth is better than investing nothing. 

You can also try our  Valuation Modeling Course by   clicking the banner below now!

Everything You Need To Master Valuation Modeling

To Help You Thrive in the Most Prestigious Jobs on Wall Street.

Researched and authored by Anushka Raj Sonkar | LinkedIn

Free Resources

To continue learning and advancing your career, check out these additional helpful  WSO  resources:

• Bull Put Spread
• Buying on Margin
• Open-end vs. Closed-end Mutual Funds
• Option Greeks
• Quantitative Trading

Get instant access to lessons taught by experienced private equity pros and bulge bracket investment bankers including financial statement modeling, DCF, M&A, LBO, Comps and Excel Modeling.

or Want to Sign up with your social account?

What is a scientific hypothesis?

It's the initial building block in the scientific method.

Hypothesis basics

What makes a hypothesis testable.

• Types of hypotheses
• Hypothesis versus theory

Additional resources

Bibliography.

A scientific hypothesis is a tentative, testable explanation for a phenomenon in the natural world. It's the initial building block in the scientific method . Many describe it as an "educated guess" based on prior knowledge and observation. While this is true, a hypothesis is more informed than a guess. While an "educated guess" suggests a random prediction based on a person's expertise, developing a hypothesis requires active observation and background research. 

The basic idea of a hypothesis is that there is no predetermined outcome. For a solution to be termed a scientific hypothesis, it has to be an idea that can be supported or refuted through carefully crafted experimentation or observation. This concept, called falsifiability and testability, was advanced in the mid-20th century by Austrian-British philosopher Karl Popper in his famous book "The Logic of Scientific Discovery" (Routledge, 1959).

A key function of a hypothesis is to derive predictions about the results of future experiments and then perform those experiments to see whether they support the predictions.

A hypothesis is usually written in the form of an if-then statement, which gives a possibility (if) and explains what may happen because of the possibility (then). The statement could also include "may," according to California State University, Bakersfield .

Here are some examples of hypothesis statements:

• If garlic repels fleas, then a dog that is given garlic every day will not get fleas.
• If sugar causes cavities, then people who eat a lot of candy may be more prone to cavities.
• If ultraviolet light can damage the eyes, then maybe this light can cause blindness.

A useful hypothesis should be testable and falsifiable. That means that it should be possible to prove it wrong. A theory that can't be proved wrong is nonscientific, according to Karl Popper's 1963 book " Conjectures and Refutations ."

An example of an untestable statement is, "Dogs are better than cats." That's because the definition of "better" is vague and subjective. However, an untestable statement can be reworded to make it testable. For example, the previous statement could be changed to this: "Owning a dog is associated with higher levels of physical fitness than owning a cat." With this statement, the researcher can take measures of physical fitness from dog and cat owners and compare the two.

Types of scientific hypotheses

In an experiment, researchers generally state their hypotheses in two ways. The null hypothesis predicts that there will be no relationship between the variables tested, or no difference between the experimental groups. The alternative hypothesis predicts the opposite: that there will be a difference between the experimental groups. This is usually the hypothesis scientists are most interested in, according to the University of Miami .

For example, a null hypothesis might state, "There will be no difference in the rate of muscle growth between people who take a protein supplement and people who don't." The alternative hypothesis would state, "There will be a difference in the rate of muscle growth between people who take a protein supplement and people who don't."

If the results of the experiment show a relationship between the variables, then the null hypothesis has been rejected in favor of the alternative hypothesis, according to the book " Research Methods in Psychology " (​​BCcampus, 2015). 

There are other ways to describe an alternative hypothesis. The alternative hypothesis above does not specify a direction of the effect, only that there will be a difference between the two groups. That type of prediction is called a two-tailed hypothesis. If a hypothesis specifies a certain direction — for example, that people who take a protein supplement will gain more muscle than people who don't — it is called a one-tailed hypothesis, according to William M. K. Trochim , a professor of Policy Analysis and Management at Cornell University.

Sometimes, errors take place during an experiment. These errors can happen in one of two ways. A type I error is when the null hypothesis is rejected when it is true. This is also known as a false positive. A type II error occurs when the null hypothesis is not rejected when it is false. This is also known as a false negative, according to the University of California, Berkeley . 

A hypothesis can be rejected or modified, but it can never be proved correct 100% of the time. For example, a scientist can form a hypothesis stating that if a certain type of tomato has a gene for red pigment, that type of tomato will be red. During research, the scientist then finds that each tomato of this type is red. Though the findings confirm the hypothesis, there may be a tomato of that type somewhere in the world that isn't red. Thus, the hypothesis is true, but it may not be true 100% of the time.

Scientific theory vs. scientific hypothesis

The best hypotheses are simple. They deal with a relatively narrow set of phenomena. But theories are broader; they generally combine multiple hypotheses into a general explanation for a wide range of phenomena, according to the University of California, Berkeley . For example, a hypothesis might state, "If animals adapt to suit their environments, then birds that live on islands with lots of seeds to eat will have differently shaped beaks than birds that live on islands with lots of insects to eat." After testing many hypotheses like these, Charles Darwin formulated an overarching theory: the theory of evolution by natural selection.

"Theories are the ways that we make sense of what we observe in the natural world," Tanner said. "Theories are structures of ideas that explain and interpret facts." 

• Read more about writing a hypothesis, from the American Medical Writers Association.
• Find out why a hypothesis isn't always necessary in science, from The American Biology Teacher.
• Learn about null and alternative hypotheses, from Prof. Essa on YouTube .

Encyclopedia Britannica. Scientific Hypothesis. Jan. 13, 2022. https://www.britannica.com/science/scientific-hypothesis

Karl Popper, "The Logic of Scientific Discovery," Routledge, 1959.

California State University, Bakersfield, "Formatting a testable hypothesis." https://www.csub.edu/~ddodenhoff/Bio100/Bio100sp04/formattingahypothesis.htm  

Karl Popper, "Conjectures and Refutations," Routledge, 1963.

Price, P., Jhangiani, R., & Chiang, I., "Research Methods of Psychology — 2nd Canadian Edition," BCcampus, 2015.‌

University of Miami, "The Scientific Method" http://www.bio.miami.edu/dana/161/evolution/161app1_scimethod.pdf  

William M.K. Trochim, "Research Methods Knowledge Base," https://conjointly.com/kb/hypotheses-explained/  

University of California, Berkeley, "Multiple Hypothesis Testing and False Discovery Rate" https://www.stat.berkeley.edu/~hhuang/STAT141/Lecture-FDR.pdf  

University of California, Berkeley, "Science at multiple levels" https://undsci.berkeley.edu/article/0_0_0/howscienceworks_19

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

Bizarre evolutionary roots of Africa's iconic upside-down baobab trees revealed

Snake Island: The isle writhing with vipers where only Brazilian military and scientists are allowed

Newfound autoimmune syndrome tied to COVID-19 can trigger deadly lung scarring

Most Popular

• 2 'It was not a peaceful crossing': Hannibal's troops linked to devastating fire 2,200 years ago in Spain
• 3 Snake Island: The isle writhing with vipers where only Brazilian military and scientists are allowed
• 4 Newfound 'glitch' in Einstein's relativity could rewrite the rules of the universe, study suggests
• 5 Scientists prove 'quantum theory' that could lead to ultrafast magnetic computing
• 5 Alien 'Dyson sphere' megastructures could surround at least 7 stars in our galaxy, new studies suggest

• Privacy Policy

Home » What is a Hypothesis – Types, Examples and Writing Guide

What is a Hypothesis – Types, Examples and Writing Guide

Table of Contents

Definition:

Hypothesis is an educated guess or proposed explanation for a phenomenon, based on some initial observations or data. It is a tentative statement that can be tested and potentially proven or disproven through further investigation and experimentation.

Hypothesis is often used in scientific research to guide the design of experiments and the collection and analysis of data. It is an essential element of the scientific method, as it allows researchers to make predictions about the outcome of their experiments and to test those predictions to determine their accuracy.

Types of Hypothesis

Types of Hypothesis are as follows:

Research Hypothesis

A research hypothesis is a statement that predicts a relationship between variables. It is usually formulated as a specific statement that can be tested through research, and it is often used in scientific research to guide the design of experiments.

Null Hypothesis

The null hypothesis is a statement that assumes there is no significant difference or relationship between variables. It is often used as a starting point for testing the research hypothesis, and if the results of the study reject the null hypothesis, it suggests that there is a significant difference or relationship between variables.

Alternative Hypothesis

An alternative hypothesis is a statement that assumes there is a significant difference or relationship between variables. It is often used as an alternative to the null hypothesis and is tested against the null hypothesis to determine which statement is more accurate.

Directional Hypothesis

A directional hypothesis is a statement that predicts the direction of the relationship between variables. For example, a researcher might predict that increasing the amount of exercise will result in a decrease in body weight.

Non-directional Hypothesis

A non-directional hypothesis is a statement that predicts the relationship between variables but does not specify the direction. For example, a researcher might predict that there is a relationship between the amount of exercise and body weight, but they do not specify whether increasing or decreasing exercise will affect body weight.

Statistical Hypothesis

A statistical hypothesis is a statement that assumes a particular statistical model or distribution for the data. It is often used in statistical analysis to test the significance of a particular result.

Composite Hypothesis

A composite hypothesis is a statement that assumes more than one condition or outcome. It can be divided into several sub-hypotheses, each of which represents a different possible outcome.

Empirical Hypothesis

An empirical hypothesis is a statement that is based on observed phenomena or data. It is often used in scientific research to develop theories or models that explain the observed phenomena.

Simple Hypothesis

A simple hypothesis is a statement that assumes only one outcome or condition. It is often used in scientific research to test a single variable or factor.

Complex Hypothesis

A complex hypothesis is a statement that assumes multiple outcomes or conditions. It is often used in scientific research to test the effects of multiple variables or factors on a particular outcome.

Applications of Hypothesis

Hypotheses are used in various fields to guide research and make predictions about the outcomes of experiments or observations. Here are some examples of how hypotheses are applied in different fields:

• Science : In scientific research, hypotheses are used to test the validity of theories and models that explain natural phenomena. For example, a hypothesis might be formulated to test the effects of a particular variable on a natural system, such as the effects of climate change on an ecosystem.
• Medicine : In medical research, hypotheses are used to test the effectiveness of treatments and therapies for specific conditions. For example, a hypothesis might be formulated to test the effects of a new drug on a particular disease.
• Psychology : In psychology, hypotheses are used to test theories and models of human behavior and cognition. For example, a hypothesis might be formulated to test the effects of a particular stimulus on the brain or behavior.
• Sociology : In sociology, hypotheses are used to test theories and models of social phenomena, such as the effects of social structures or institutions on human behavior. For example, a hypothesis might be formulated to test the effects of income inequality on crime rates.
• Business : In business research, hypotheses are used to test the validity of theories and models that explain business phenomena, such as consumer behavior or market trends. For example, a hypothesis might be formulated to test the effects of a new marketing campaign on consumer buying behavior.
• Engineering : In engineering, hypotheses are used to test the effectiveness of new technologies or designs. For example, a hypothesis might be formulated to test the efficiency of a new solar panel design.

How to write a Hypothesis

Here are the steps to follow when writing a hypothesis:

Identify the Research Question

The first step is to identify the research question that you want to answer through your study. This question should be clear, specific, and focused. It should be something that can be investigated empirically and that has some relevance or significance in the field.

Conduct a Literature Review

Before writing your hypothesis, it’s essential to conduct a thorough literature review to understand what is already known about the topic. This will help you to identify the research gap and formulate a hypothesis that builds on existing knowledge.

Determine the Variables

The next step is to identify the variables involved in the research question. A variable is any characteristic or factor that can vary or change. There are two types of variables: independent and dependent. The independent variable is the one that is manipulated or changed by the researcher, while the dependent variable is the one that is measured or observed as a result of the independent variable.

Formulate the Hypothesis

Based on the research question and the variables involved, you can now formulate your hypothesis. A hypothesis should be a clear and concise statement that predicts the relationship between the variables. It should be testable through empirical research and based on existing theory or evidence.

Write the Null Hypothesis

The null hypothesis is the opposite of the alternative hypothesis, which is the hypothesis that you are testing. The null hypothesis states that there is no significant difference or relationship between the variables. It is important to write the null hypothesis because it allows you to compare your results with what would be expected by chance.

Refine the Hypothesis

After formulating the hypothesis, it’s important to refine it and make it more precise. This may involve clarifying the variables, specifying the direction of the relationship, or making the hypothesis more testable.

Examples of Hypothesis

Here are a few examples of hypotheses in different fields:

• Psychology : “Increased exposure to violent video games leads to increased aggressive behavior in adolescents.”
• Biology : “Higher levels of carbon dioxide in the atmosphere will lead to increased plant growth.”
• Sociology : “Individuals who grow up in households with higher socioeconomic status will have higher levels of education and income as adults.”
• Education : “Implementing a new teaching method will result in higher student achievement scores.”
• Marketing : “Customers who receive a personalized email will be more likely to make a purchase than those who receive a generic email.”
• Physics : “An increase in temperature will cause an increase in the volume of a gas, assuming all other variables remain constant.”
• Medicine : “Consuming a diet high in saturated fats will increase the risk of developing heart disease.”

Purpose of Hypothesis

The purpose of a hypothesis is to provide a testable explanation for an observed phenomenon or a prediction of a future outcome based on existing knowledge or theories. A hypothesis is an essential part of the scientific method and helps to guide the research process by providing a clear focus for investigation. It enables scientists to design experiments or studies to gather evidence and data that can support or refute the proposed explanation or prediction.

The formulation of a hypothesis is based on existing knowledge, observations, and theories, and it should be specific, testable, and falsifiable. A specific hypothesis helps to define the research question, which is important in the research process as it guides the selection of an appropriate research design and methodology. Testability of the hypothesis means that it can be proven or disproven through empirical data collection and analysis. Falsifiability means that the hypothesis should be formulated in such a way that it can be proven wrong if it is incorrect.

In addition to guiding the research process, the testing of hypotheses can lead to new discoveries and advancements in scientific knowledge. When a hypothesis is supported by the data, it can be used to develop new theories or models to explain the observed phenomenon. When a hypothesis is not supported by the data, it can help to refine existing theories or prompt the development of new hypotheses to explain the phenomenon.

When to use Hypothesis

Here are some common situations in which hypotheses are used:

• In scientific research , hypotheses are used to guide the design of experiments and to help researchers make predictions about the outcomes of those experiments.
• In social science research , hypotheses are used to test theories about human behavior, social relationships, and other phenomena.
• I n business , hypotheses can be used to guide decisions about marketing, product development, and other areas. For example, a hypothesis might be that a new product will sell well in a particular market, and this hypothesis can be tested through market research.

Characteristics of Hypothesis

Here are some common characteristics of a hypothesis:

• Testable : A hypothesis must be able to be tested through observation or experimentation. This means that it must be possible to collect data that will either support or refute the hypothesis.
• Falsifiable : A hypothesis must be able to be proven false if it is not supported by the data. If a hypothesis cannot be falsified, then it is not a scientific hypothesis.
• Clear and concise : A hypothesis should be stated in a clear and concise manner so that it can be easily understood and tested.
• Based on existing knowledge : A hypothesis should be based on existing knowledge and research in the field. It should not be based on personal beliefs or opinions.
• Specific : A hypothesis should be specific in terms of the variables being tested and the predicted outcome. This will help to ensure that the research is focused and well-designed.
• Tentative: A hypothesis is a tentative statement or assumption that requires further testing and evidence to be confirmed or refuted. It is not a final conclusion or assertion.
• Relevant : A hypothesis should be relevant to the research question or problem being studied. It should address a gap in knowledge or provide a new perspective on the issue.

Advantages of Hypothesis

Hypotheses have several advantages in scientific research and experimentation:

• Guides research: A hypothesis provides a clear and specific direction for research. It helps to focus the research question, select appropriate methods and variables, and interpret the results.
• Predictive powe r: A hypothesis makes predictions about the outcome of research, which can be tested through experimentation. This allows researchers to evaluate the validity of the hypothesis and make new discoveries.
• Facilitates communication: A hypothesis provides a common language and framework for scientists to communicate with one another about their research. This helps to facilitate the exchange of ideas and promotes collaboration.
• Efficient use of resources: A hypothesis helps researchers to use their time, resources, and funding efficiently by directing them towards specific research questions and methods that are most likely to yield results.
• Provides a basis for further research: A hypothesis that is supported by data provides a basis for further research and exploration. It can lead to new hypotheses, theories, and discoveries.
• Increases objectivity: A hypothesis can help to increase objectivity in research by providing a clear and specific framework for testing and interpreting results. This can reduce bias and increase the reliability of research findings.

Limitations of Hypothesis

Some Limitations of the Hypothesis are as follows:

• Limited to observable phenomena: Hypotheses are limited to observable phenomena and cannot account for unobservable or intangible factors. This means that some research questions may not be amenable to hypothesis testing.
• May be inaccurate or incomplete: Hypotheses are based on existing knowledge and research, which may be incomplete or inaccurate. This can lead to flawed hypotheses and erroneous conclusions.
• May be biased: Hypotheses may be biased by the researcher’s own beliefs, values, or assumptions. This can lead to selective interpretation of data and a lack of objectivity in research.
• Cannot prove causation: A hypothesis can only show a correlation between variables, but it cannot prove causation. This requires further experimentation and analysis.
• Limited to specific contexts: Hypotheses are limited to specific contexts and may not be generalizable to other situations or populations. This means that results may not be applicable in other contexts or may require further testing.
• May be affected by chance : Hypotheses may be affected by chance or random variation, which can obscure or distort the true relationship between variables.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

• Table of Contents
• Random Entry
• Chronological
• Editorial Information
• About the SEP
• Editorial Board
• How to Cite the SEP
• Special Characters
• Advanced Tools
• Support the SEP
• PDFs for SEP Friends
• Make a Donation
• SEPIA for Libraries
• Entry Contents

Bibliography

Academic tools.

• Friends PDF Preview
• Author and Citation Info
• Back to Top

Chance versus Randomness

Randomness, as we ordinarily think of it, exists when some outcomes occur haphazardly, unpredictably, or by chance. These latter three notions are all distinct, but all have some kind of close connection to probability. Notoriously, there are many kinds of probability: subjective probabilities (‘degrees of belief’), evidential probabilities, and objective chances, to name a few (Hájek 2012), and we might enquire into the connections between randomness and any of these species of probability. In this entry, we focus on the potential connections between randomness and chance, or physical probability. The ordinary way that the word ‘random’ gets used is more or less interchangeable with ‘chancy’, which suggests this Commonplace Thesis—a useful claim to target in our discussion:

The Commonplace Thesis, and the close connection between randomness and chance it proposes, appears also to be endorsed in the scientific literature, as in this example from a popular textbook on evolution (which also throws in the notion of unpredictability for good measure):

scientists use chance, or randomness, to mean that when physical causes can result in any of several outcomes, we cannot predict what the outcome will be in any particular case. (Futuyma 2005: 225)

Some philosophers are, no doubt, equally subject to this unthinking elision, but others connect chance and randomness deliberately. Suppes approvingly introduces

the view that the universe is essentially probabilistic in character, or, to put it in more colloquial language, that the world is full of random happenings. (Suppes 1984: 27)

However a number of technical and philosophical advances in our understanding of both chance and randomness open up the possibility that the easy slide between chance and randomness in ordinary and scientific usage—a slide that would be vindicated by the truth of the Commonplace Thesis—is quite misleading. This entry will attempt to spell out these developments and clarify the differences between chance and randomness, as well as the areas in which they overlap in application. It will also aim to clarify the relationship of chance and randomness to other important notions in the vicinity, particularly determinism and predictability (themselves often subject to confusion).

There will be philosophically significant consequences if the Commonplace Thesis is incorrect, and if ordinary usage is misleading. For example, it is intuitively plausible that if an event is truly random it cannot be explained (if it happens for a reason, it isn’t truly random). It might seem then that the possibility of probabilistic explanation is undermined when the probabilities involved are genuine chances. Yet this pessimistic conclusion only follows under the assumption, derived from the Commonplace Thesis, that all chancy outcomes are random. Another interesting case is the role of random sampling in statistical inference. If randomness requires chance, then no statistical inferences on the basis of ‘randomly’ sampling a large population will be valid unless the experimental design involves genuine chance in the selection of subjects. But the rationale for random sampling may not require chance sampling—as long as our sample is representative, those statistical inferences may be reliable. But in that case, we’d be in a curious situation where random sampling wouldn’t have much to do with randomness, and whatever justification for beliefs based on random sampling that randomness is currently thought to provide would need to be replaced by something else.

A final case of considerable philosophical interest is the frequentist approach to objective probability, which claims (roughly) that the chance of an outcome is its frequency in an appropriate series of outcomes (Hájek 2012 §3.4). To avoid classifying perfectly regular recurring outcomes as chancy, frequentists like von Mises (1957) proposed to require that the series of outcomes should be random, without pattern or order. Frequentism may fall with the Commonplace Thesis: if there can be chancy outcomes without randomness, both will fail.

The Commonplace Thesis is central to all three examples. As it is widely accepted that probabilistic explanation is legimitate, that random sampling doesn’t need genuine chance (though it can help), and that frequentism is in serious trouble (Hájek 1997), there is already some some pressure on the Commonplace Thesis. But we must subject it to closer examination to clarify whether these arguments do succeed, and what exactly it means to say of some event or process that it is random or chancy. Though developing further consequences of this kind is not the primary aim of this entry, it is hoped that what is said here may help to untangle these and other vexed issues surrounding chance and randomness.

1.1 ‘Single-Case’ Chance and Chance Processes

1.2 physics and chance, 2.1.1 randomness and gambling systems—von mises’ account, 2.1.2 randomness and effective tests: martin-löf-randomness, 2.2.1 kolmogorov complexity and randomness, 2.2.2 prefix-free kolmogorov complexity, 2.3 schnorr’s theorem: kolmogorov and ml-randomness coincide, 3. the commonplace thesis refined, 4.1 unrepresentative outcome sequences, 4.2 the reference class problem, 4.4 dependence: randomness is indifferent to history, 4.5 pseudorandom sequences, 5.1 short sequences, 5.2 chaotic dynamics, 5.3 classical indeterminism, 5.4 drawing without replacement, 6.1 product chance, 6.2 process randomness: epistemic theories, 7.1 chance and determinism, 7.2 randomness and determinism, 8. conclusion, other internet resources, related entries.

To get clear on the connections and differences between chance and randomness, it would be good first to have some idea of what chance and randomness amount to. Interestingly, philosophical attention has focussed far more on chance than randomness. This may well be a consequence of the Commonplace Thesis. Whatever its source, we can appeal to a substantial consensus in the philosophical literature as to what kind of thing chance must be.

Carnap (1945) distinguished between two conceptions of probability, arguing that both were scientifically important. His ‘probability\(_1\)’ corresponds to an epistemic notion, nowadays glossed either (following Carnap himself) as evidential probability, or as credence or degree of belief. This is contrasted with Carnap’s ‘probability\(_2\)’, which is the concept of a non-epistemic objective kind of probability, better known as chance .

There are many philosophical accounts of what actually grounds chance, as part of the minor philosophical industry of producing ‘interpretations’—really, reductive analyses or explications—of probability. In this vein we have at least the frequency theory of Reichenbach (1949) and von Mises (1957) (and Carnap’s own explication of probability\(_2\) was in terms of frequencies), the propensity theory of Popper (1959) and Giere (1973), as well as many more recent accounts, notably Lewis’ (1994) ‘Best System’ account of chance (see also Loewer 2004). There is no agreement over which, if any, of these accounts are right; certainly both the accounts mentioned face difficulties in giving an adequate account of chance. The consensus mentioned earlier is not over what actually plays the role of chance, but rather on the constraints that determine what that role is.

There can be such a consensus because ‘chance’ is not a technical term, but is rather an ordinary concept deployed in fairly familiar situations (games of chance, complicated and unpredictable scenarios, large ensembles of similar events, etc.). There is widespread agreement amongst native speakers of English over when ‘chance’ applies to a particular case, and this agreement at least indicates that there is a considerable body of ordinary belief about chance. One needn’t take the deliverances of folk intuition as sacrosanct to recognise that this ordinary belief provides the starting point for philosophical accounts of chance. It may turn out that nothing fits the role picked out by these ordinary beliefs and their philosophical precisifications, yet even in that case we’d be inclined to conclude that chance doesn’t exist, not that our ordinary beliefs about what chance must be are incorrect.

Below, some of the theoretical principles that philosophers have extracted from commonplace beliefs about chance will be outlined. (In so doing, we freely use probabilistic notation and concepts; see the entry on interpretations of probability, Hájek 2012 : §1 , for background probability theory required to understand these expressions.) Two such constraints have been widely accepted since the early days of the philosophy of probability. Firstly, it is required that the mathematics of chance should conform to some standard mathematical theory of probability such as Kolmogorov’s 1933 axiomatisation of the probability calculus (or some recognisable variant thereof, like Popper’s axiomatisation of conditional probability). Secondly, chance should be objective: mind-independent and not epistemic or evidential. But a number of other constraints have been articulated and defended in the literature. (Schaffer 2007: §4 contains a useful discussion of these and other constraints on the chance role.) While these principles have been termed ‘things we know about chance’, this shouldn’t be taken to preclude our discovering that there is no such thing as chance—rather, the philosophical consensus is that if there is any such thing as chance, it will (more or less) fit these constraints.

Chance should regulate (that is, it is involved with norms governing) rational belief in line with Lewis’ Principal Principle (Lewis, 1980), or his New Principle (Lewis, 1994; Hall, 2004). Where \(C\) is a reasonable initial credence function, and \(E\) is the evidence, the Principal Principle (omitting some complications) is this:

(PP) \(C(p \mid \ulcorner\Ch(p) = x\urcorner \wedge E) = x\);

This principle says that rational initial credence should treat chance as an expert, deferring to it with respect to opinions about the outcome \(p\), by adopting the corresponding chances as your own conditional degree of belief. The New Principle—adopted to deal with some problematic interactions between Lewis’ metaphysics and the PP—advocates deferring to chance in a somewhat different way. This New Principle NP suggests (more or less) that rational initial credence should treat chance as an expert with respect to the outcome \(p\), by adopting the conditional chances, on the admissible evidence, as your conditional degrees of belief on that same evidence, as in this principle (this claim corresponds to equation 3.9 in Hall 2004; see his discussion at pp. 102–5 for some important qualifications, and the connection to the formulation of the NP in Lewis 1994):

(Chance-Analyst) \(C(p \mid \ulcorner\Ch(p \mid E) = x\urcorner \wedge E) = x\).

Non-reductionist views about chance, which take chances to be independent fundamental features of reality, can follow PP. Reductionists, who take the values of chances to be fixed entirely by other features of reality (normally frequencies and symmetries, but chance is typically constrained by them in a quite indirect manner), are for technical reasons (to do with undermining, see supplement A.1 ) ordinarily forced to adopt NP, and Chance-Analyst, as norms on credence, though in many ordinary cases, NP and PP give very similar recommendations. In either case, both formal principles give content to the intuitively plausible idea that chances should guide reasonable credence. [ 1 ]

Chance should connect with possibility. Leibniz claimed that probability was a kind of ‘graded possibility’, and more recent authors have largely agreed. In particular, it seems clear that if an outcome has some chance of occurring, then it is possible that the outcome occurs. This intuition has been made precise in the Basic Chance Principle (BCP) (See supplement A.2 for further details on this principle):

Suppose \(x \gt 0\) and \(Ch_{tw}(A) = x\). Then \(A\) is true in at least one of those worlds \(w'\) that matches \(w\) up to time \(t\) and for which \(Ch_t (A) = x\). (Bigelow et al. , 1993: 459)

But one needn’t accept precisely this version of the BCP to endorse the general thesis that chance and possibility must be linked—for other versions of this kind of claim, see Mellor (2000); Eagle (2011) and the ‘realization principle’ of Schaffer (2007: 124).

Chance should connect with actual frequencies, at least to the extent of permitting frequencies to be good evidence for the values of chances. This may be through some direct connection between chance and frequency, or indirectly through the influence of observed outcome frequencies on credences about chances via the Principal Principle (Lewis, 1980: 104–6). But chance should not be identified with frequency—since a fair coin can produce any sequence of outcomes, there is no possibility of identifying chance with observed frequency. (Though of course a fair coin is overwhelmingly likely to produce roughly even numbers of heads and tails when tossed often enough). Moreover, there can be a chance for a kind of outcome even when there are very few instances of the relevant process that leads to that outcome, resulting in the actual frequencies being misleading or trivial (for example, if there is only a single actual outcome: Hájek 1997).

When considering the connection between frequency and chance, not just any frequency will do. What is wanted is the frequency in relevantly similar trials, with the same kind of experimental setup. The relevance of frequencies in such trial is derived from the assumption that in such similar trials, the same chances exist: intra-world duplicate trials should have the same chances. This is closely related to the ‘stable trial principle’ (Schaffer, 2003: 37ff). Chances attach to the outcomes of trials, but the physical grounds of the chance lie in the physical properties of the trial device or chance setup.

More details on all of these principles can be found in this supplementary document:

Supplement A. Some Basic Principles About Chance

Chance, it is commonly said, is ‘single-case objective probability’. Philosophers haven’t been very clear on what is meant by ‘single-case’, and the terminology is slightly misleading, as it falsely suggests that perhaps multiple cases have less claim to their chances. The most minimal version of the claim is that, at least sometimes, an outcome can have a chance to be a result of an instance of a given kind of process, or trial , even though no other trials of that process occur. This is what we will mean by ‘single case’ chance. (A stronger claim is that the chance of an outcome resulting from a given process is an intrinsic property of a single trial. The stronger claim is inconsistent with standard versions of the frequency theory, and indeed it can be difficult to see how chance and frequency might be connected if that stronger claim were true.) Some have claimed that single-case chance is no part of objective probability; for example, von Mises (1957: p. 11) remarks that the ‘concept of probability … applies only to problems in which either the same event repeats itself again and again, or a great number of uniform elements are involved at the same time.’. However, this is a theoretical judgment on von Mises’ part, based on difficulties he perceived in giving an account of single-case chance; it is not a judgement derived from internal constraints on the chance role. And how could it be? For

like it or not, we have this concept [of single-case chance]. We think that a coin about to be tossed has a certain chance of falling heads, or that a radioactive atom has a certain chance of decaying within the year, quite regardless of what anyone may believe about it and quite regardless of whether there are any other similar coins or atoms. As philosophers we may well find the concept of objective chance troublesome, but that is no excuse to deny its existence, its legitimacy, or its indispensability. If we can’t understand it, so much the worse for us. (Lewis, 1980: 90)

A number of the constraints discussed above require the legitimacy of single-case chance. The objection to frequentist accounts of chance that the frequency might misrepresent the chance if the number of actual outcomes is too low apparently requires that there be non-trivial chances even for events resulting from a type of trial that occurs only very few times, and perhaps even can only occur very few times (Hájek 2009: 227–8). The strong connections between possibility and chance mooted by the BCP and variants thereof also require that there are single-case chances. For the BCP requires that, for every event with some chance, it is possible that the event has that same chance and occurs. As noted in Supplement A.2 , this renders the chances relatively independent of the occurrent frequencies, which in turn requires single-case chance. For some single outcomes—for example, the next toss of a coin that is biased ⅔ towards heads—only very few assignments of credence are reasonable; in that case, we should, if we are rational, have credence of ⅔ in the coin landing heads. The rationality of this unequal assignment cannot be explained by anything like symmetry or indifference. Its rationality can be explained by the PP only if the single-case chance of heads on the next toss is ⅔. Moreover, the existence of this constraint on rational credence should have an explanation. Therefore the PP, if it is to play this necessary explanatory role, requires single-case chance.

It is the stable trial principle that has the closest connection with single-case chance, however. For in requiring that duplicate trials should receive the same chances, it is natural to take the chance to be grounded in the properties of that trial, plus the laws of nature. It is quite conceivable that the same laws could obtain even if that kind of trial has only one instance, and the very same chances should be assigned in that situation. But then there are well-defined chances even though that type of event occurs only once.

The upshot of this discussion is that chance is a process notion, rather than being entirely determined by features of the outcome to which the surface grammar of chance ascriptions assigns the chance. For if there can be a single-case chance of \(\frac{1}{2}\) for a coin to land heads on a toss even if there is only one actual toss, and it lands tails, then surely the chance cannot be fixed by properties of the outcome ‘lands heads’, as that outcome does not exist. [ 2 ] The chance must rather be grounded in features of the process that can produce the outcome: the coin-tossing trial, including the mass distribution of the coin and the details of how it is tossed, in this case, plus the background conditions and laws that govern the trial. Whether or not an event happens by chance is a feature of the process that produced it, not the event itself. The fact that a coin lands heads does not fix that the coin landed heads by chance, because if it was simply placed heads up, as opposed to tossed in a normal fashion, we have the same outcome not by chance. Sometimes features of the outcome event cannot be readily separated from the features of its causes that characterise the process by means of which it was produced. But the upshot of the present discussion is that even in those cases, whether an outcome happens by chance is fixed by the properties of the process leading up to it, the causal situation in which it occurs, and not simply by the fact that an event of a given character was the product of that process. [ 3 ]

Do chances exist? The best examples of probability functions that meet the principles about chance are those provided by our best physical theories. In particular, the probability functions that feature in radioactive decay and quantum mechanics have some claim to being chance functions. In orthodox approaches to quantum mechanics, some measurements of a system in a given state will not yield a result that represents a definite feature of that prior state (Albert 1992). So, for example, an \(x\)-spin measurement on a system in a determinate \(y\)-spin state will not yield a determinate result reflecting some prior state of \(x\)-spin, but rather has a 0.5 probability of resulting in \(x\)-spin \(= +1\), and a 0.5 probability of resulting in \(x\)-spin \(= -1\). That these measurement results cannot reflect any prior condition of the system is a consequence of various no-hidden variables theorems, the most famous of which is Bell’s theorem (Bell 1964; see the entry on Bell’s theorem, Shimony 2009 ). Bell’s theorem shows that the probabilities predicted by quantum mechanics, and experimentally confirmed, for spin measurements on a two-particle entangled but spatially separated system cannot be equal to the joint probabilities of two independent one-particle systems. The upshot is that the entangled system cannot be represented as the product of two independent localised systems with determinate prior \(x\)-spin states. Therefore, there can be no orthodox local account of these probabilities of measurement outcomes as reflecting our ignorance of a hidden quality found in half of the systems, so that the probabilities are in fact basic features of the quantum mechanical systems themselves. [ 4 ]

The standard way of understanding this is that something—the process of measurement, on the Copenhagen interpretation, or spontaneous collapse on the GRW theory—induces a non-deterministic state transition, called collapse , into a state in which the system really is in a determinate state with respect to a given quality (though it was not previously). These transition probabilities are dictated entirely by the state and the process of collapse, which allows these probabilities to meet the stable trial principle. The models of standard quantum mechanics explicitly permit two systems prepared in identical states to evolve via collapse into any state which has a non-zero prior probability in the original state, which permits these probabilities to meet the BCP. And the no-hidden variables theorems strongly suggest that there is no better information about the system to guide credence in future states than the chances, which makes these probabilities play the right role in the PP. These basic quantum probabilities governing state transitions seem to be strong candidates to be called chances.

The foregoing argument makes essential use of collapse. The existence of collapse as an alternative rule governing the evolution of the quantum state controversial, and it is a scandal of quantum mechanics that we have no satisfactory understanding of why collapse (or measurement) should give rise to basic probabilities. But that said, the existence of well-confirmed probabilistic theories which cannot be plausibly reduced to any non-probabilistic theory is some evidence that there are chances. (Though the Everettian (‘many-worlds’) program of generating quantum probabilities from subjective uncertainty, without basic chance in the present sense, has recently been gaining adherents—see Barrett 1999; Wallace 2007.) Indeed, it looks like the strongest kind of evidence that there are chances. For if our best physical theories did not feature probabilities, we should have little reason to postulate them, and little reason to take chances to exist. This will become important below (§ 5 ), when we discuss classical physics. The conventional view of classical physics, including statistical mechanics, is that it does not involve basic probability (because the state transition dynamics is deterministic), and is not accordingly a theory that posits chances (Loewer 2001). [ 5 ] Below, we will examine this view, as well as some of the recent challenges to this conventional view. But there is at least enough evidence from fundamental physics for the existence of chances for us to adopt it already at this point as a defensible assumption.

2. Randomness

As mentioned in the introduction, some philosophers deliberately use ‘random’ to mean ‘chancy’. A random process, in their view, is one governed by chance in the sense of the previous section. This generates stipulative definitions like this one:

I group random with stochastic or chancy, taking a random process to be one which does not operate wholly capriciously or haphazardly but in accord with stochastic or probabilistic laws. (Earman 1986: 137)

This process conception of randomness is perfectly legitimate, if somewhat redundant. But it is not adequate for our purposes. It makes the Commonplace Thesis a triviality, and thereby neither interesting in itself nor apt to support the interesting conclusions some have drawn from it concerning explanation or experimental design. Moreover,

The invocation of a notion of process randomness is inadequate in another way, as it does not cover all cases of randomness. Take a clear case of process randomness, such as one thousand consecutive tosses of a fair coin. We would expect, with very high confidence, to toss at least one head. But as that outcome has some chance of not coming to pass, it counts as process random even when it does. This is at variance with what we would ordinarily say about such an outcome, which is not at all unexpected, haphazard, or unpredictable. We could search for some refinement of the notion of process randomness that would reserve the word ‘random’ for more irregular looking outcomes. But a better approach, and the one we pursue in this entry, is to distinguish between randomness of the process generating an outcome (which we stipulate to amount to its being a chance process), and randomness of the product of that random process. In the case just envisaged, we have a random process, while the outomce ‘at least one head in 1000 tosses’ is not a random product.

The introduction of product randomness helps us make sense of some familiar uses of ‘random’ to characterise an entire collection of outcomes of a given repeated process. This is the sense in which a random sample is random: it is an unbiased representation of the population from which it is drawn—and that is a property of the entire sample, not each individual member. If a random sample is to do its job, it should be irregular and haphazard with respect to the population variables of interest. We should not be able to predict the membership of the sample to any degree of reliability by making use of some other feature of individuals in the population. (So we should not be able to guess at the likely membership of a random sample by using some feature like ‘is over 180cm tall’.) A random sample is one that is representative in the sense of being typical of the underlying population from which it is drawn, which means in turn that—in the ideal case—it will exhibit no order or pattern that is not exemplified in that underlying population.

While many random samples will be drawn using a random process, they need not be. For example, if we are antecedently convinced that the final digit of someone’s minute of birth is not correlated with their family income, we may draw a random sample of people’s incomes by choosing those whose birth minute ends in ‘7’, and that process of choice is not at all random. To be sure that our sample is random, we may wish to use random numbers to decide whether to include a given individual in the sample; to that end, large tables of random digits have been produced, displaying no order or pattern (RAND Corporation 1955). This other conception of randomness, as attaching primarily to collections of outcomes, has been termed product randomness .

Product randomness also plays an important role in scientific inference. Suppose we encounter a novel phenomenon, and attempt to give a theory of it. All we have to begin with is the data concerning what happened. If that data is highly regular and patterned, we may attempt to give a deterministic theory of the phenomenon. But if the data is irregular and disorderly—random—we may offer only a stochastic theory. As we cannot rely on knowing whether the phenomenon is chancy in advance of developing a theory of it, it is extremely important to be able to characterise whether the data is random or not directly, without detouring through prior knowledge of the process behind it. We might think that we could simply do this by examination of the data—surely the lack of pattern will be apparent to the observer? (We may assume that patternlessness is good evidence for randomness, even if not entailed by it.) Yet psychological research has repeatedly shown that humans are poor at discerning patterns, seeing them in completely random data, and (for precisely the same reason, in fact) failing to see them in non-random data (Gilovich et al. , 1985; Kahneman and Tversky, 1972; Bar-Hillel and Wagenaar, 1991; Hahn and Warren, 2009). So the need for an objective account of randomness of a sequence of outcomes is necessary for reliable scientific inference.

It might seem initially that giving a rigorous characterisation of disorder and patternlessness is a hopeless task, made even more difficult by the fact that we need to characterise it without using the notion of chance. (Otherwise we make CT trivial.) Yet a series of mathematical developments in the theory of algorithmic randomness , culminating in the early 1970s, showed that a satisfactory characterisation of randomness of a sequence of outcomes was possible. This notion has shown its theoretical fruitfulness not only in the foundations of statistics and scientific inference, but also in connection with the development of information theory and complexity theory. The task of this section is to introduce the mathematical approach to the definition of random sequences, just as we introduced the philosophical consensus on chance in the previous section. We will then be in a position to evaluate the Commonplace Thesis, when made precise using theoretically fruitful notions of chance and randomness.

The fascinating mathematics of algorithmic randomness are largely unknown to philosophers. For this reason, I will give a fairly detailed exposition in this entry. Various points of more technical interest have been relegated to this supplementary document:

Supplement B. Further Details Concerning Algorithmic Randomness

Most proofs will be skipped, or relegated to this supplementary document:

Supplement C. Proofs of Selected Theorems

Fuller discussions can be found in the cited references.

Throughout the focus will be on a simple binary process, which has only two types of outcome \(O = \{0,1\}\). (The theory of randomness for the outcome sequences of such a simple process can be extended to more complicated sets of outcomes, but there is much of interest even in the question which binary sequences are product random? ) A sequence of outcomes is an ordered collection of events, finite or infinite, such that each event is of a type in \(O\). So a sequence \(x = x_1 x_2 \ldots x_k\ldots\), where each \(x_i \in O\). The set of all infinite binary sequences of outcomes is known as the Cantor space . One familiar example of a process the outcomes of which form a Cantor space is an infinite sequence of independent flips of a fair coin, where 1 denotes heads and 0 tails. Notions from measure theory and computability theory are used in the discussion below; an elementary presentation of the mathematics needed can be found in supplement B.2 .

2.1 Product Randomness: Random Sequences are Most Likely

Perhaps counterintuitively, we begin with the case of infinite binary sequences. Which of these should count as random products of our binary process? Each individual infinite sequence, whether orderly or not, has measure zero under the standard (Lebesgue) measure over the Cantor space. We cannot determine whether an individual sequence is random from considering what fraction it constitutes of the set of all such sequences. But, intuitively, almost all such infinite sequences should be random and disorderly, and only few will be orderly (an observation first due to Ville 1939). A typical infinite sequence is one without pattern; only exceptional cases have order to them. If the actual process that generate the sequences are perfectly deterministic, it may be that a typical product of that process is not random. But we are rather concerned to characterise which of all the possible sequences produced by any process whatsoever are random, and it seems clear that most of the ways an infinite sequence might be produced, and hence most of the sequences so produced, will be random. This fits with intuitive considerations:

We arrange in our thought, all possible events in various classes; and we regard as extraordinary those classes which include a very small number. In the game of heads and tails, if heads comes up a hundred times in a row, then this appears to us extraordinary, because the almost infinite number of combinations that can arise in a hundred throws are divided in regular sequences, or those in which we observe a rule that is easy to grasp, and in irregular sequences, that are incomparably more numerous. (Laplace 1826)

This fertile remark underscores both that random sequences should be unruly , and that they should be common . In the present framework: the set of non -random sequences should have measure zero, proportional to the set of all such sequences—correspondingly, the set of random sequences should have measure one (Dasgupta, 2011: §3; Gaifman and Snir, 1982: 534; Williams, 2008: 407–11).

This helps, but not much. For there are many measure one subsets of the Cantor space, and we need some non-arbitrary way of selecting a privileged such subset. (The natural option, to take the intersection of all measure one subsets, fails, because the complement of the singleton of any specific sequence is measure one, so for each sequence there is a measure one set which excludes it; therefore the intersection of all measure one sets excludes every sequence, so is the empty set.) The usual response is to take the random sequences to be the intersection of all measure one subsets of the space which have ‘nice’ properties, and to give some principled delimitation of which properties are to count as ‘nice’ and why.

For example, if a sequence is genuinely random, we should expect that in the long run it would tend to have features we associate with the outputs of (independent, identically distributed trials of) a chancy process. The sequence should look as disorderly as if it were the expected product of genuine chance. This approach is known accordingly as the typicality approach to randomness. Typicality is normally defined with respect to a prior probability function, since what is a typical series of fair coin toss outcomes might not be a typical series of unfair coin toss outcomes (Eagle 2016: 447). In the present case, we use the Lebesgue measure as it is the natural measure definable from the symmetries of the outcome space of the binary process itself.

A typical sequence should satisfy all of the various ‘properties of stochasticity’ (Martin-Löf 1966: 604). What are these properties? They include the property of large numbers, the claim that the limit frequency of a digit in a random sequence should not be biased to any particular digit. The (strong) law of large numbers is the claim that, with probability 1, an infinite sequence of independent, identically distributed Bernoulli trials will have the property of large numbers. If we concentrate on the sequence of outcomes as independently given mathematical entities, rather than as the products of a large number of independent Bernoulli trials, we can follow Borel’s (1909) characterisation of the strong law. Let \(S_n (x)\) be the number of 1s occurring in the first \(n\) places of sequence \(x\) (this is just \(\sum^{n}_{k=1}x_{k}\)), and let \(B\) be the set of infinite sequences \(x\) such that the limit of \(S_n (x)/n\) as \(n\) tends to infinity is \(\frac{1}{2}\). Borel’s theorem is that \(B\) has measure one; almost all infinite sequences are, in the limit, unbiased with respect to digit frequency.

Clearly, the property of large numbers is a necessary condition for randomness of a sequence. It is not sufficient, however. Consider the sequence 10101010…. This sequence is not biased. But it is clearly not random either, as it develops in a completely regular and predictable fashion. So we need to impose additional constraints. Each of these constraints will be another property of stochasticity we should expect of a random sequence, including all other such limit properties of ‘unbiasedness’.

One such further property is Borel normality , also defined in that paper by Borel. A sequence is Borel normal iff each finite string of digits of equal length has equal frequency in the sequence. [ 6 ] Borel proved that a measure one set of sequences in the Cantor space are Borel normal. Borel normality is a useful condition to impose for random sequences, as it has the consequence that there will be no predictable pattern to the sequence: for any string \(\sigma\) appearing multiple times in a sequence, it will as commonly be followed by a 1 as by a 0. This lack of predictability based on the previous elements of the sequence is necessary for genuine randomness. But again Borel normality is not sufficient for randomness. The Champernowne sequence (Champernowne 1933) is the sequence of digits in the binary representations of each successive non-negative integer:

This is Borel normal, but perfectly predictable, because there is a general law which states what the value of the sequence at each index will be—not because it can be predicted from prior elements of the sequence, but because it can be predicted from the index.

We must impose another condition to rule out the Champernowne sequence. We could proceed, piecemeal, in response to various problem cases, to successively introduce further stochastic properties, each of which is a necessary condition for randomness, eventually hoping to give a characterisation of the random sequences by aggregating enough of them together. Given the complex structure of the Cantor space, the prospects for success of such a cumulative approach seem dim. A more promising bolder route is to offer one stochastic property that is by itself necessary and sufficient for randomness, the possession of which will entail the possession of the other properties we’ve mentioned (the property of large numbers, Borel normality, etc.).

The first detailed and sophisticated attempt at a bolder approach to defining randomness for a sequence with a single stochastic property was by von Mises (von Mises, 1957; von Mises, 1941). Suppose you were presented with any subsequence \(x_1 , \ldots ,x_{n-1}\) of (not necessarily consecutive members of) a sequence, and asked to predict the value of \(x_n\). If the sequence were really random, then this information—the values of any previous members of the sequence, and the place of the desired outcome in the sequence—should be of no use to you in this task. To suppose otherwise is to suppose that there is an exploitable regularity in the random sequence; a gambler could, for example, bet reliably on their preferred outcome and be assured of a positive expected gain if they were in possession of this information. A gambling system selects points in a sequence of outcomes to bet on; a successful gambling system would be one where the points selected have a higher frequency of ‘successes’ than in the sequence as a whole, so that by employing the system one can expect to do better than chance. But the failure of gambling systems to make headway in games of chance suggests that genuinely random sequences of outcomes aren’t so exploitable. Von Mises, observing the empirical non-existence of successful gambling systems, makes it a condition of randomness for infinite sequences that they could not be exploited by a gambling system (his ‘ Prinzip vom ausgeschlossenen Spielsystem ’). The idea is that without what effectively amounts to a crystal ball, there is no way of selecting a biased selection of members of a random sequence.

Shorn of the inessential presentational device of selecting an outcome to bet on based on past outcomes, von Mises contends that it is a property of stochasticity that a random sequence should not be such that information about any initial subsequence \(x_1 x_2 \ldots x_{k-1}\) provides information about the contents of outcome \(x_k\). He formally implements this idea by defining a place selection as ‘the selection of a partial sequence in such a way that we decide whether an element should or should not be included without making use of the [value] of the element’ (von Mises 1957: 25). He then defines a random sequence as one such that every infinite subsequence selected by an admissible place selection retains the same relative digit frequencies as in the original sequence (so one cannot select a biased subsequence, indicating that this is a genuine property of stochasticity). In our case this will mean that every admissibly selected subsequence will meet the property of large numbers with equal frequency of 1s and 0s. [ 7 ] One way to characterise the resulting set of von Mises-random (vM-random) sequences is that it is the largest set which contains only infinite sequences with the right limit frequency and is closed under all admissible place selections. If the limit frequency of a digit is 1, say in the sequence \(111\ldots\), it is true that every admissible place selection determines a subsequence with the same limit frequency. Von Mises intends this result, for this is what a random sequence of outcomes of trials with probability 1 of obtaining the outcome 1 looks like. This sequence does not meet the property of large numbers, however. So we modify von Mises’ own condition, defining the vM-random sequences as the largest set of infinite sequences which have the limit frequency \(\frac{1}{2}\), and which is closed under all admissible place selections.

Von Mises’ original proposals were deliberately imprecise about what kinds of procedures count as admissible place selections. This imprecision did not concern him, as he was disposed to regard the ‘right’ set of place selections for any particular random collective as being fixed by context and not invariantly specifiable. But his explicit characterisation is subject to counterexamples. Since ‘any increasing sequence of natural numbers \(n_1 \lt n_2 \lt \ldots\) defines a corresponding selection rule, … given an arbitrary sequence of 0s and 1s … there is among the selection rules the one which selects the 1s of the given sequence, so the limit frequency is changed’ (Martin-Löf 1969b: 27). This clearly violates von Mises’ intentions, as he presumably intended that the place selections should be constructively specified, yet the notion of vM-randomness remains tantalisingly vague without some more concrete specification.

Such a specification arrived in the work of Church (1940), drawing on the then newly clarified notion of an effective procedure. Church observed that

To a player who would beat the wheel at roulette a system is unusable which corresponds to a mathematical function known to exist but not given by explicit definition; and even the explicit definition is of no use unless it provides a means of calculating the particular values of the function. … Thus a [gambling system] should be represented mathematically, not as a function, or even as a definition of a function, but as an effective algorithm for the calculation of the values of a function. (Church 1940: 133)

Church therefore imposes the condition that the admissible place selections should be, not arbitrary functions, but effectively computable functions of the preceding outcomes in a sequence. Formally, we consider (following Wald 1938) a place selection as a function \(f\) from an initial segment \(x_1 x_2 \ldots x_{i-1}\) of a sequence \(\sigma\) into \(\{0,1\}\), such that the selected subsequence \(\sigma' = \{x_i : f(x_1 \ldots x_{i - 1}) = 1\}\), Church’s proposal is that we admit only those place selections which are computable (total recursive) functions. [ 8 ] Church’s proposal applies equally to the sequences von Mises is concerned with, namely those with arbitrary non-zero outcome frequencies for each type of outcome; to get the random sequences, we again make the restriction to those normal binary sequences where the limit relative frequency of each outcome is \(\frac{1}{2}\) (these are sometimes called the Church stochastic sequences).

As Church points out, if we adopt the Church-Turing computable functions as the admissible place selections, it follows quickly that the set of admissible place selections is countably infinite. We may then show:

Theorem 1 (Doob-Wald). The set of random sequences forms a measure one subset of the Cantor space. [ Proof ]

Thus von Mises’ conception of randomness was made mathematically robust (Martin-Löf 1969b). We can see that various properties of stochasticity follow from this characterisation. For example, we can show:

Corollary 1 . Every von Mises-random sequence is Borel normal. [ Proof ]

These successes for the approach to randomness based on the impossibility of gambling systems were, however, undermined by a theorem of Ville (1939):

Theorem 2 (Ville). For any countable set of place selections \(\{\phi_n\}\) (including the identity), there exists an infinite binary sequence \(x\) such that: for all \(m\), \(\lim_{m \rightarrow \infty} (\sum^{m}_{k=1}(\phi_n (x))_k)/m = \frac{1}{2}\); but for all \(m, (\sum^{m}_{n=1}x_{n})/m \gt \frac{1}{2}\).

That is, for any specifiable set of place selections, including the total recursive place selections proposed by Church as the invariantly appropriate set, there exist sequences which have the right limit relative frequencies to satisfy the strong law of large numbers (and indeed Borel normality), as do all their acceptable subsequences, but which are biased in all initial segments. [ 9 ]

Why should this be a problem for random sequences? The property of large numbers shows that almost all infinite binary sequences have limit digit frequency \(\frac{1}{2}\), but says nothing about how quickly this convergence happens or about the statistical properties of the initial segments. There certainly exist sequences that converge to \(\frac{1}{2}\) but where \(S_n (x)/n \gt \frac{1}{2}\) for all \(n\) (the sequence has the right mean but converges ‘from above’). In the ‘random walk’ model of our random sequences, where each element in the sequence is interpreted as a step left (if 0) or right (if 1) along the integer line, this sequence would consist of a walk that (in the limit) ends up back at the origin but always (or even eventually) stays to the right. Intuitively, such a sequence is not random.

Such sequences do indeed violate at least one property of stochasticity, as it turns out that in a measure one set of sequences, \(S_n (x)/n\) will be above the mean infinitely many times, and below the mean infinitely many times. So stated, this is the law of symmetric oscillation (Dasgupta, 2011: 13). [ 10 ] Since the law of symmetric oscillations holds for a measure one set of sequences, it is a plausible property of randomness (it is naturally of a family with other properties of stochasticity). Ville’s result shows that von Mises’ definition in terms of place selections cannot characterise random sequences exactly because it includes sequences that violate this law (so don’t correspond to a truly random random walk). Indeed, such sequences don’t even correspond to von Mises’ avowed aims. As Li and Vitányi say (2008: 54), ‘if you bet 1 all the time against such a sequence of outcomes, your accumulated gain is always positive’. As such, Ville-style sequences seem to permit successful gambling, despite the fact that they do not permit a system to be formulated in terms of place selections.

Von Mises and Church identified a class of sequences, those with limit frequencies invariant under recursive place selections, that satisfied a number of the measure one stochastic properties of sequences that are thought characteristic of randomness. But the class they identified was too inclusive. The next insight in this area was due to (Martin-Löf 1966), who realised that rather than looking for another single property of sequences that would entail that the sequence met all the further conditions on randomness, it was simpler to adopt as a definition that a sequence is random iff the sequence has all the measure one properties of randomness that can be specified. Here again recursion theory plays a role, for the idea of a measure one property of randomness that can be specified is equivalent to the requirement that there be an effective procedure for testing whether a sequence violates the property. This prompts the following very bold approach to the definition of random sequences:

Martin-Löf Randomness : A random sequence is one which cannot be effectively determined to violate a measure one randomness property (Downey and Hirschfeldt 2010: §5.2; Dasgupta 2011: §6.1; Porter 2016: 461–2).

Recalling the definition of effective measure zero from supplement B.2 , Martin-Löf suggests that a random sequence is any that does not belong to any effective measure zero set of sequences, and thus belongs to every effective measure one set of sequences. An effective measure zero set of sequences will contain sequences which can be effectively determined to have a ‘special hallmark’ (for example, having a ‘1’ at every seventh place, or never having the string ‘000000’ occurring as a subsequence). It is part of von Mises’ insight that no random sequence will possess any of these effectively determinable special hallmarks: such hallmarks would permit exploitation as part of a gambling system. But Martin-Löf notices that all of the commonly used measure one properties of stochasticity are effective measure one. Any sequence which violates the property of large numbers, or the law of symmetric oscillations, etc., will do so on increasingly long initial subsequences. So the violation of any such property will also be a special hallmark of a non-random sequence, an indicator that the sequence which possesses it is an unusual one. Since the unusual properties of non-stochasticity in question are effective measure zero, we can therefore say that the random sequences are those which are not special in any effectively determinable way. To formalise this, Martin-Löf appeals to the language of significance testing. His main result is sometimes put as the claim that random sequences as those which pass all recursive significance tests for sequences (Schnorr 1971: §1)—they are never atypical enough to prompt us to reject the hypothesis that they are random. See supplement B.1.1 for more detail on this point.

Note that the restriction to effective properties of sequences is crucial here. If we allowed, for example, the property being identical to my favourite random sequence x , that would define a test which the sequence \(x\) would fail, even though it is random. But it follows from our observations about von Mises randomness (which is still a necessary condition on randomness) that no effectively computable sequence is random (if it were, there would be a place selection definable from the algorithm that selected all the 1s in the sequence). So there is no effective test that checks whether a given sequence is identical to some random sequence.

The central result of Martin-Löf (1966) is the following:

Theorem 3 (Universal Tests and the Existence of Random Sequences). There is a universal test for ML-randomness; moreover, only a measure zero set of infinite binary sequences fails this test. So almost all such sequences are ML-random. [ Proof ]

The universal test does define an effective measure one property, but (unlike normality, or having no biased admissible subsequences), it is far from a naturally graspable property. Nevertheless, Martin-Löf’s result does establish that there are random sequences that satisfy all the properties of stochasticity, and that in fact almost all binary sequences are random in that sense. Returning to Ville’s theorem 2 , it can be shown that all ML-random sequences satisfy the law of symmetric oscillations (van Lambalgen 1987a: §3.3). Hence the kind of construction Ville uses yields vM-random sequences which are not ML-random. All the ML-random sequences have the right limit relative frequencies, since they satisfy the effective measure one property of large numbers. So Martin-Löf random sequences satisfy all the intuitive properties we would expect of a sequence produced by tosses of a fair coin, but are characterised entirely by reference to the effectively specifiable measure one sets of infinite sequences. We have therefore characterised random sequences entirely in terms of the explicit features of the product, and not of the process that may or may not lie behind the production of these sequences.

There are other accounts that develop and extend Martin-Löf’s account of randomness, in the same kind of framework, such as that of Schnorr (1971); for some further details, see supplement B.1.2 .

2.2 Product Randomness: Random Sequences are Most Disorderly

For infinite binary sequences, the Martin-Löf definition in terms of effective tests is a robust and mathematically attractive notion. However, it seems to have the major flaw that it applies only to infinite binary sequences. (Since finiteness of a sequence is effectively positively decidable, and the set of all finite sequences is measure zero, every finite sequence violates an effective measure one randomness property.) Yet ordinarily we are happy to characterise even quite small finite sequences of outcomes as random. As mentioned above (§ 2 ), there is room for doubt at our ability to do so correctly, as we seem to be prone to mischaracterise sequences we are presented with, and perform poorly when asked to produce our own random sequences. However, there is nothing in this literature to suggest that we are fundamentally mistaken in applying the notion of randomness to finite sequences at all. So one might think this shows that the Martin-Löf approach is too restrictive.

Yet there is something in the idea of ML-randomness that we might apply profitably to the case of finite sequences. Since being generated by an effective procedure is a measure zero property of infinite sequences, given that there are only countably many effective procedures, it follows immediately that no ML-random sequence can be effectively produced. This fits well with the intuitive idea that random sequences don’t have the kind of regular patterns that any finite algorithm, no matter how complex, must exploit in order to produce an infinite sequence. This contrast between random sequences which lack patterns that enable them to be algorithmic generated, and non-random sequences which do exhibit such patterns, does not apply straightforwardly to the finite case, because clearly there is an effective procedure which enables us to produce any particular finite sequence of outcomes—simply to list those outcomes in the specification of the algorithm. But a related contrast does exist—between those algorithms which are simply crude lists of outcomes, and those which produce outcomes which involve patterns and regularities in the outcome sequence. This leads us to the idea that finite random sequences, like their infinite cousins, are not able to be generated by an algorithm which exploits patterns in the outcome sequence. The outcomes in random sequences are thus patternless, or disorderly , in a way that is intuitively characteristic of random sequences.

Disorderly sequences, in the above sense, are highly incompressible . The best effective description we can give of such a sequence—one that would enable someone else, or a computer, to reliably reproduce it—would be to simply list the sequence itself. This feature allows us to characterise the random sequences as those which cannot be produced by a compact algorithm (compact with respect to the length of the target sequence, that is). Given that algorithms can be specified by a list of Turing machine instructions, we have some basic idea on how to characterise the length of an algorithm. We can then say that a random sequence is one such that the shortest algorithm which produces it is approximately (to be explained below) the same length as the sequence itself—no greater compression in the algorithm can be attained. This proposal, suggested by the work of Kolmogorov, Chaitin and Solomonov (KCS), characterises randomness as the algorithmic or informational complexity of a sequence. Comprehensive surveys of complexity and the complexity-based approach to randomness are Li and Vitányi 2008 and Downey and Hirschfeldt 2010: Part I. (See also Chaitin 1975, Dasgupta 2011: §7, Downey et al. 2006: §§1–3, Earman 1986: 141–7, Kolmogorov 1963, Kolmogorov and Uspensky 1988, Smith (1998: ch. 9) and van Lambalgen 1995.)

If \(f\) is effectively computable—a recursive function—let us say that \(\delta\) is an \(f\)-description of a finite string \(\sigma\) iff \(f\) yields \(\sigma\) on input \(\delta\). We may define the \(f\) -complexity of a string \(\sigma , C_f (\sigma)\), as the length of the shortest string \(\delta\) that \(f\)-describes \(\sigma\). If there is no such \(\delta\), let the \(f\)-complexity of \(\sigma\) be infinite. \(f\) is thus a decompression algorithm, taking the compressed description \(\delta\) back to the original string \(\sigma\). There are obviously many different kinds of decompression algorithm. One boring case is the identity function (the empty program), which takes each string to itself. The existence of this function shows that there are decompression algorithms \(f\) which have a finite \(f\)-complexity for any finite string. Any useful decompression algorithm will, however, yield an output string significantly longer than the input description, for at least some input descriptions.

One example is this algorithm: on input of a binary string \(\delta\) of length \(4n\), the algorithm breaks the input down into \(n\) blocks of 4, which it turns into an output sequence \(\sigma\), as follows. Given a block \(b_1 , \ldots ,b_4\), it produces a block of the symbol contained in \(b_1\), the length of which is governed by the binary numeral \(b_2 b_3 b_4\). So the block 1101 produces a string of five 1s. The output sequence is obtained by concatenating the output of successive blocks in order. Every string \(\sigma\) can be represented by this algorithm, since the string \(\sigma '\) which involves replacing every 1 in \(\sigma\) by 1001, and every 0 by 0001, will yield \(\sigma\) when given as input to this algorithm. So this algorithm has finite complexity for any string. But this algorithm can do considerably better; if the original string, for example, is a string of sixteen 1s, it can be obtained by input of this description: 11111111, which is half the length. Indeed, as reflection on this algorithm shows, this algorithm can reconstruct an original string from a shorter description, for many strings, particularly if they contain reasonably long substrings of consecutive 1s or 0s.

However, there is a limit on how well any algorithm can compress a string. If \(\lvert\sigma\rvert\) is the length of \(\sigma\), say that a string \(\sigma\) is compressed by \(f\) if there is an \(f\)-description \(\delta\) such that \(\sigma = f(\delta)\) and \(\lvert\delta\rvert \lt \lvert\sigma\rvert\). If a useful decompression algorithm is such that for some fixed \(k\), \(\lvert f(\delta)\rvert \le \lvert\delta\rvert + k\), so that \(f\)-descriptions are at least \(k\) shorter than the sequence to be compressed, then it follows that very few strings usefully compress. For there are \(2^l\) strings \(\sigma\) such that \(\lvert\sigma\rvert = l\); so there are at most \(2^{l - k} f\)-descriptions; since \(f\) is a function, there are at most \(2^{l-k}\) compressible strings. As a proportion of all strings of length \(l\), then, there are at most \(2^{l-k}/2^{l} = \frac{1}{2}^{k}\) compressible strings . This means that as the amount of compression required increases, the number of sequences so compressible decreases exponentially. Even in the most pitiful amount of compression, \(k = 1\), we see that at most half the strings of a given length can be compressed by any algorithm \(f\).

So our interest must be in those decompression functions which do best overall. We might hope to say: \(f\) is better than \(g\) iff for all \(\sigma , C_f (\sigma) \le C_g (\sigma)\). Unfortunately, no function is best in this sense, since for any given string \(\sigma\) with \(f\)-complexity \(\lvert\sigma\rvert - k\), we can design a function \(g\) as follows: on input 1, output \(\sigma\); on input n\(\delta\) (for any \(n)\), output \(f(\delta)\). (Generalising, we can add arbitrarily long prefixes of length \(m\) onto the inputs to \(g\) and have better-than\(-f\) compression for \(2^m\) sequences.) But we can define a notion of complexity near-superiority of \(f\) to \(g\) iff there is some constant \(k\) such that for any string, \(C_f (\sigma) \le C_g (\sigma) + k. f\) is least as good as \(g\), subject to some constant which is independent of the functions in question. If \(f\) and \(g\) are both complexity near-superior to each other, for the same \(k\), we say they are complexity equivalent .

Kolmogorov (1965) showed that there is an optimal decompression algorithm:

Theorem 4 (Kolmogorov). There exists a decompression algorithm which is near-superior to any other program. Moreover, any such optimal algorithm is complexity equivalent to any other optimal algorithm (see also Chaitin 1966 and Martin-Löf 1969a). [ Proof ]

Such a universal function Kolmogorov called asymptotically optimal (for as \(\lvert\sigma\rvert\) increases, the constant \(k\) becomes asymptotically negligible).

Choose some such asymptotically optimal function \(u\), and define the complexity (simpliciter) \(C(\sigma) = C_u (\sigma)\). Since \(u\) is optimal, it is near-superior to the identity function; it follows that there exists a \(k\) such that \(C(\sigma) \le \lvert\sigma\rvert + k\). On the other hand, we also know that the number of strings for which \(C(\sigma) \le \lvert\sigma\rvert - k\) is at most \(\frac{1}{2} ^k\). We know therefore that all except \(1 - 2^k\) sequences of length \(n\) have a complexity within \(k\) of \(n\). As \(n\) increases, for fixed large \(k\), therefore, we see that almost all sequences have complexity of approximately their length. All this can be used to make precise the definition of randomness sketched above.

Kolmogorov random : We say that a sequence \(\sigma\) is Kolmogorov-random iff \(C(\sigma) \approx \lvert\sigma\rvert\).

It follows from what we have just said that there exist random sequences for any chosen length \(n\), and that as \(n\) increases with respect to \(k\), random sequences come to be the overwhelming majority of sequences of that length.

Theorem 5. A random sequence of a given length cannot be effectively produced. [ Proof ]

An immediate corollary is that the complexity function \(C\) is not a recursive function. If it were, for any \(n\), we could effectively compute \(C(\sigma)\) for any \(\sigma\) of length \(n\). By simply listing all such sequences, we could halt after finding the first \(\sigma\) for which \(C(\sigma) \ge n\). But then we could effectively produce a random sequence, contrary to theorem 5 .

The notion of Kolmogorov randomness fits well with the intuitions about the disorderliness of random sequences we derived from the Martin-Löf account. It also fits well with other intuitions about randomness—random sequences don’t have a short description, so there is no sense in which they are produced in accordance with a plan. As such, Kolmogorov randomness also supports von Mises’ intuitions about randomness being linked to the impossibility of gambling systems, as there will be no way of effectively producing a given random sequence of outcomes using a set of initially given data any smaller than the sequence itself. There is no way of predicting a genuinely random sequence in advance because no random sequence can be effectively produced, yet every predictable sequence of outcomes can (intuitively) be generated by specifying the way in which future outcomes can be predicted on the basis of prior outcomes. Moreover, because for increasing \(k\) the number of strings of length \(n\) which are random increases, and because for increasing \(n\) we can choose larger and larger \(k\), there is some sense in which the great majority of sequences are random; this matches well the desiderata in the infinite case that almost all sequences should be random. Finally, it can be shown that the Kolmogorov randomness of a sequence is equivalent to that sequence passing a battery of statistical tests, in the Martin-Löf sense—indeed, that the Kolmogorov random sequences are just those that pass a certain universal test of non-randomness (Martin-Löf 1969a: §2). [ 11 ]

The plain Kolmogorov complexity measure is intuitively appealing. Yet the bewildering variety of permitted \(f\)-descriptions includes many unmanageable encodings. In particular, for a given decompression algorithm \(f\), there are \(f\)-descriptions \(\gamma\) and \(\delta\) such that \(\delta = \gamma\unicode{x2040}\tau\), for some string \(\tau\). This is an inefficient encoding, because if \(\gamma\) can occur both as a code itself, and as the initial part of another code, then an algorithm cannot decode its input string ‘on the fly’ as soon as it detects a comprehensible input, but must wait until it has scanned and processed the entire input before beginning to decode it. An efficient coding, such that no acceptable input is an initial substring of another acceptable input, is called prefix-free (because no member is a prefix of any other member). A good example of an encoding like this is the encoding of telephone numbers: the telephone exchange can, on input of a string of digits that it recognises, immediately connect you; once an acceptable code from a prefix-free set has been input, no other acceptable code can follow it.

Prefix-free encodings are useful for a number of practical purposes, and they turn out to be useful in defining randomness also. (As we will see in § 2.3 , they are of special importance in avoiding a problem in the definition of infinite Kolmogorov random sequences.) The change is the natural one: we appeal, not to the plain complexity of a sequence in defining its randomness, but the so-called prefix-free complexity (Downey and Hirschfeldt 2010: §2.5ff; Dasgupta 2011: §8).

To fix ideas, it is useful to have an example of a prefix-free encoding in mind. Suppose we have a string \(\sigma =x_1 \ldots x_k\) of length \(k\). This is the initial part of the string \(\sigma 1\), so if any string was an acceptable input, we would not have a prefix-free encoding. But if the code contained information about the length of the string encoded, we would know that the length of \(\sigma , k\), is less than the length of \(\sigma 1\). We can make this idea precise as follows (using a code similar to that used to a different end in the proof of Theorem 4 ). Let the code of \(\sigma\) be the string \(1^{[\lvert\sigma\rvert]}0\sigma\)—that is, the code of a string consists of a representation of the length of the string, followed by a 0, followed by the string. This is clearly a prefix-free encoding. [ 12 ] This coding is not particularly efficient, but more compact prefix-free encodings do exist.

The notion of prefix-free complexity is defined in exactly the same way as plain complexity, with the additional restriction that the allowable \(f\)-descriptions of a string, given a decompression function \(f\), must form a prefix-free set. With an appropriate choice of coding, we can get a set of \(f\)-descriptions which is monotonic with increasing length, i.e., if \(\lvert\gamma\rvert \lt \lvert\delta\rvert\) then \(\lvert f(\gamma)\rvert \lt \lvert f(\delta)\rvert\). Our definition goes much as before: The prefix free Kolmogorov complexity, given a decompression function \(f\) with a prefix-free domain, of a string \(\sigma\), denoted \(K_f (\sigma)\), is the length of the shortest \(f\)-description of \(\sigma\) (and infinite otherwise). Since \(1^{[\lvert\sigma\rvert]}0\sigma\) is a finite prefix-free code for \(\sigma\), we know there are at least some prefix-free decompression algorithms with finite \(K_f\) for every string. As before, we can show there exist better decompression algorithms than this one, and indeed, that there exists a universal prefix-free decompression algorithm \(u\), such that for every other algorithm \(m\) there is a \(k\) such that \(K_u (\sigma) \le K_m (\sigma) + k\), for all \(\sigma\) (Downey and Hirschfeldt 2010: §2.5). We define \(K(\sigma) = K_u (\sigma)\).

Since the set of prefix-free codes is a subset of the set of all possible codes, we should expect generally that \(C(\sigma) \le K(\sigma)\). On the other hand, we can construct a universal prefix-free algorithm \(u\) as follows. A universal Turing machine \(u'\) takes as input the Gödel number of the Turing machine we wish to simulate, and the input we wish to give to that machine. Let us concatenate these two inputs into a longer input string that is itself uniquely readable; and we then encode that longer string into our prefix-free encoding. The encoding is effectively computable, clearly, so we can chain a decoding machine together with our universal machine \(u'\); on input an acceptable prefix-free string, the decoder will decompose it into the input string, we then decompose the input string into the Gödel number and the input, and run the machine \(u'\) on that pair of inputs. Depending on the particular encoding we choose, we may establish various bounds on \(K\); one obvious bound we have already established is that \(C(\sigma) \le K(\sigma) \lt C(\sigma) + 2\lvert\sigma\rvert\). By using a more efficient prefix-free coding of a \(u'\)-description, we can establish better bounds. [ 13 ] (Some more results on the connection between \(K\) and \(C\) are in Downey and Hirschfeldt 2010: §§3.1–3.2.)

With prefix-free complexity in hand, we may define:

Prefix-free Kolmogorov Randomness : A string \(\sigma\) is Prefix-free Kolmogorov random iff \(K(\sigma) \ge \lvert\sigma\rvert\) (modulo an additive constant).

Again, there do exist prefix-free random sequences, since we know that there are plain random sequences, and given the greater length of a prefix-free encoding, we know that the prefix-free code of ordinary random sequence will be generally longer than an arbitrary code of it, and thus random too. Indeed, there will be more prefix-free random sequences because strings compress less effectively under \(K\) than \(C\). Yet \(K\) and \(C\) behave similarly enough that the success of plain Kolmogorov complexity at capturing our intuitions about randomness carry over to prefix-free Kolmogorov randomness, and the label ‘Kolmogorov random’ has come to be used generally to refer to prefix-free Kolmogorov random sequences.

Both plain and prefix-free Kolmogorov randomness provide satisfactory accounts of the randomness of finite sequences. One difficulty arises, as I suggested earlier, when we attempt to extend plain Kolmogorov randomness to the case of infinite sequences in the most obvious way, that is, by defining an infinite sequence as Kolmogorov random iff all finite initial segments are Kolmogorov random. It would then turn out that no infinite sequence is random. Why? Because of the following theorem, which shows that there is no sequence such that all of its initial segments are random:

Theorem 6 (Martin-Löf 1966) . For any sufficiently long string, there will always be some fairly compressible initial segments. (See also Li and Vitányi 2008: §2.5.1 and Downey and Hirschfeldt 2010: §2.1.) [ Proof ]

This dip in complexity of an initial subsequence will occur infinitely often in even a random infinite sequence, a phenomenon known as complexity oscillation (Li and Vitányi 2008: §2.5.1). This phenomenon means that ‘it is difficult to express a universal sequential test precisely in terms of \(C\)-complexity’ (Li and Vitányi 2008: 151), and the best that can be precisely done is to find upper and lower bounds expressible in terms of ordinary Kolmogorov complexity between which the set of ML-random sequences falls (Li and Vitányi 2008: § 2.5.3).

However, the phenomenon of complexity oscillation does not pose as significant a problem for prefix-free Kolmogorov complexity. Complexity oscillation does arise, but in fact the inefficiency of prefix-free encodings is a benefit here: ‘\(K\) exceeds \(C\) by so much that the complexity of the prefix does not drop below the length of the prefix itself (for random infinite \(\omega)\)’ (Li and Vitányi 2008: 221). That is, while the complexity of some initial segments dips down, it always remains greater than the length of the prefix. So it can be that, uniformly, when \(x\) is an infinite sequence, for any of its initial subsequences \(\sigma , K(\sigma) \ge \lvert\sigma\rvert\). This suggests that we can extend prefix-free Kolmogorov complexity to the infinite case in the straightforward way: an infinite sequence \(x\) is prefix-free Kolmogorov random iff every finite initial subsequence is prefix-free Kolmogorov random.

With this definition in hand we obtain a very striking result. The class of infinite prefix-free Kolmogorov random sequences is certainly non-empty. Indeed: it is just the class of ML-random sequences!

Theorem 7 (Schnorr) . A sequence is ML-random iff it is prefix-free Kolmogorov random. [ Proof ]

Schnorr’s theorem is evidence that we really have captured the intuitive notion of randomness. Different intuitive starting points have generated the same set of random sequences. This has been taken to be evidence that ML-randomness or equivalently (prefix-free) Kolmogorov randomness is really the intuitive notion of randomness, in much the same way as the coincidence of Turing machines, Post machines, and recursive functions was taken to be evidence for Church’s Thesis , the claim that any one of these notions captures the intuitive notion of effective computability. Accordingly, Delahaye (1993) has proposed the Martin-Löf-Chaitin Thesis , that either of these definitions captures the intuitive notion of randomness. If this thesis is true, this undermines at least some sceptical contentions about randomness, such as the claim of Howson and Urbach (1993: 324) that ‘it seems highly doubtful that there is anything like a unique notion of randomness there to be explicated’.

There are some reasons to be suspicious of the Martin-Löf-Chaitin Thesis, despite the mathematically elegant convergence between these two mathematical notions. For one, there is quite a bit of intuitive support for accounts of randomness which do not make it primarily a property of sequences, and those other accounts are no less able to be made mathematically rigorous (see especially the ‘epistemic’ theories of randomness discussed in § 6.2 , as well as theories of randomness as indeterminism discussed in § 7.2 ). The existence of other intuitive notions makes the case of randomness rather unlike the supposedly analogous case of Church’s Thesis, where no robust alternative characterisation of effective computability is available.

Even if we accept that randomness, like disorder, is at root a product notion, there are a number of candidates in the vicinity of the set identified by Schnorr’s thesis that might also deserve to be called the set of random sequences. Most obviously, there is Schnorr’s own conception of randomness (§ 2.1.2 ; supplement B.1.2 ). Schnorr (1971) suggests that, for technical and conceptual reasons, Schnorr randomness is to be preferred to Martin-Löf randomness as an account of the intuitive notion. While results that parallel the convergence of ML-randomness and Kolmogorov randomness have been given (Downey and Griffiths 2004), the relevant compressibility notion of randomness for Schnorr randomness was not known until quite recently, and is certainly less intuitively clear than Kolmogorov randomness. Moreover, since the set of ML-random sequences is a strict subset of the set of Schnorr random sequences, any problematic members of the former are equally problematic members of the latter; and of course there will be Schnorr random sequences which fail some Martin-Löf statistical test, which might lead some to reject the viability of Schnorr’s notion from the start.

Schnorr’s result showing the convergence between prefix-free Kolmogorov complexity and Martin-Löf randomness is very suggestive. As has become clear, the existence of other notions of randomness—incluing Schnorr randomness, as well as a number of other proposals (Li and Vitányi 2008: §2.5; Porter 2016: 464–6)—shows that we should be somewhat cautious in yielding to its suggestion.

This is especially true in light of a recent argument by Porter (2016: 469–70). He considers a certain schematic characterisation of computable functions, something like this: for every computable function \(f\) with property \(P_i, f\) is differentiable at \(x\) iff \(x\) corresponds to a random\(_i\) sequence. It turns out that for each sense of randomness (ML randomness, Schnorr randomness, computable randomness, etc.), there is some corresponding property of computable functions. Most importantly, none of these properties look overwhelmingly more natural or canonical than the others. For example, computable functions of bounded variation are differentiable at the Martin-Löf random points, while nondecreasing computable functions are differentiable at the computably random points. The difficulty for the Martin-Löf-Chaitin Thesis is this: these results give us the notion of a typical sequence, with respect to the differentiability of various kinds of function. Unfortunately, these notions of typical sequence diverge from one another. Unlike Church’s thesis, where all the notions of effective computability line up, here we have a case where various notions of a typical sequence do not line up with each other (though there is significant overlap). Porter concludes that ‘no single definition of randomness can do the work of capturing every mathematically significant collection of typical points’ (Porter 2016: 471).

That conclusion may well be justified. But we can largely sidestep the dispute over whether there is a single precise notion of randomness that answers perfectly to our intuitive conception of random sequence. Kolmogorov-Martin-Löf randomness is a reasonable and representative exemplar of the algorithmic approach to randomness, and it overlaps almost everywhere with any other plausible definition of randomness. It is adopted here as a useful working account of randomness for sequences. None of the difficulties and problems I raise below for the connection between random sequences and chance turns in any material way on the details of which particular set of sequences gets counted as random (most are to do with the mismatch between the process notion of chance and any algorithmic conception of randomness, with differences amongst the latter being relatively unimportant). So while the observations below are intended to generalise to Schnorr randomness and other proposed definitions of random sequences, I will explicitly treat only KML randomness in what follows.

The notions of chance and randomness discussed and clarified in the previous two sections are those that have proved scientifically and philosophically most fruitful. Whatever misfit there may be between ordinary language uses of these terms and these scientific precisifications, is made up for by the usefulness of these concepts. This is particularly so with the notion of randomness, chance being in philosopher’s mouths much closer to what we ordinarily take ourselves to know about chance. On these conceptions, randomness is fundamentally a product notion, applying in the first instance to sequences of outcomes, while chance is a process notion, applying in the single case to the process or chance setup which produces a token outcome. Of course the terminology in common usage is somewhat slippery; it’s not clear, for example, whether to count random sampling as a product notion, because of the connection with randomness, or as a process notion, because sampling is a process. The orthodox view of the process, in fact, is that it should be governed by a random sequence; we enumerate the population, and sample an individual \(n\) just in case the \(n\)-th outcome in a pre-chosen random sequence is 1. (Of course the sample thus selected may not be random in some intuitive sense; nevertheless, it will not be biased because of any defect in the selection procedure, but rather only due to bad luck.)

With these precise notions in mind, we can return to the Commonplace Thesis CT connecting chance and randomness. Two readings make themselves available, depending on whether we take single outcomes or sequences of outcomes to be primary:

\(\mathbf{CTa}\): A sequence of outcomes happens by chance iff that sequence is random. \(\mathbf{CTb}\): An outcome happens by chance iff there is a random sequence of outcomes including it.

Given the standard probability calculus, any sequence of outcomes is itself an outcome (in the domain of the chance function defined over a \(\sigma\)-algebra of outcomes, as in standard mathematical probability); so we may without loss of generality consider only (CTb). But a problem presents itself, if we consider chancy outcomes in possible situations in which only very few events ever occur. It may be that the events which do occur, by chance, are all of the same type, in which case the sequence of outcomes won’t be random. This problem is analogous to the ‘problem of the single case’ for frequency views of chance ( Hájek 2012 : §3.3 ), because randomness is, like frequency, a property of an outcome sequence. The problem arises because the outcomes may be too few or too orderly to properly represent the randomness of the entire sequence of which they are part (all infinite random sequences have at least some non-random initial subsequences). The most common solution in the case of frequentism was to opt for a hypothetical outcome sequence—a sequence of outcomes produced under the same conditions with a stable limit frequency (von Mises 1957: 14–5). Likewise, we may refine the commonplace thesis as follows:

\(\mathbf{RCT}\): An outcome happens by chance iff, were the trial which generated that outcome repeated often enough under the same conditions, we would obtain a random sequence including the outcome (or of which the outcome is a subsequence).

Here the idea is that chancy outcomes would, if repeated often enough, produce an appropriately homogenous sequence of outcomes which is random. If the trial is actually repeated often enough, this sequence should be the actual sequence of outcomes; the whole point of Kolmogorov randomness was to permit finite sequences to be random.

RCT is intuitively appealing, even once we distinguish process and product randomness in the way sketched above. It receives significant support from the fact that fair coins, tossed often enough, do in our experience invariably give rise to random sequences, and that the existence of a random sequence of outcomes is compelling evidence for chance. The truth of RCT explains this useful constraint on the epistemology of chance, since if we saw an actual finite random sequence, we could infer that the outcomes constituting that sequence happened by chance. However, in the next two sections, we will see that there are apparent counterexamples even to RCT, posing grave difficulties for the Commonplace Thesis. In § 4 we will see a number of cases where there are apparently chancy outcomes without randomness, while in § 5 we will see cases of apparent randomness where there is no chance involved.

A fundamental problem with RCT seems to emerge when we consider the fate of hypothetical frequentism as a theory of chance. For there seems to be no fact of the matter, in a case of genuine chance, as to what sequence of outcomes would result: as Jeffrey (1977: 193) puts it, ‘there is no telling whether the coin would have landed head up on a toss that never takes place. That’s what probability is all about.’ Many philosophers (e.g., Hájek 2009) have followed Jeffrey in scepticism about the existence and tractability of the hypothetical sequences apparently required by the right hand side of RCT. However, there is some reason to think that RCT will fare better than hypothetical frequentism in this respect. In particular, RCT does not propose to analyse chance in terms of these hypothetical sequences, so we can rely on the law of large numbers to guide our expectation that chance processes produce certain outcome frequencies, in the limit, with probability 1; this at least may provide some reason for thinking that the outcome sequences will behave as needed for RCT to turn out true. Even so, one might suspect that many difficulties for hypothetical frequentism will recur for RCT. However, these difficulties stem from general issues with merely possible evidential manifestations of chance processes, and have nothing specifically to do with randomness. The objections canvassed below are, by contrast, specifically concerned with the interaction between chance and randomness. So these more general potential worries will be set aside, though they should not be forgotten—they may even be, in the end, the most significant problem for RCT (if the right kind of merely possible sequence doesn’t exist, we must retreat to CTa or CTb and their problems).

4. Chance Without Randomness

It is possible for a fair coin—i.e., such that the chances of heads and tails are equal—to be tossed infinitely many times, and to land heads on every toss. An infinite sequence of heads has, on the standard probability calculus, zero chance of occurring. (Indeed, it has zero chance even on most non-standard views of probability: Williamson 2007.) Nevertheless if such a sequence of outcomes did occur, it would have happened by chance—assuming, plausibly, that if each individual outcome happens by chance, the complex event composed by all of them also happens by chance. But in that case we would have an outcome that happened by chance and yet the obvious suitable sequence of outcomes is not KML-random. This kind of example exploits the fact that while random sequences are a measure one set of possible outcome sequences of any process, chancy or otherwise, measure one does not mean every .

This counterexample can be resisted. For while it is possible that a fair coin lands only heads when tossed infinitely many times, it may not be that this all heads outcome sequence is a suitable sequence. For if we consider the counterfactual involved in RCT—what would happen, if a fair coin were tossed infinitely many times—we would say: it would land heads about half the time. That is (on the standard, though not uncontroversial, Lewis-Stalnaker semantics for counterfactuals: Lewis 1973), though the all-heads outcome sequence is possible, it does not occur at any of the nearest possibilities in which a fair coin is tossed infinitely many times.

If we adopt a non-reductionist account of chance, this line of resistance is quite implausible. For there is nothing inconsistent on such views about a situation where the statistical properties of the occurrent sequence of outcomes and the chance diverge arbitrarily far, and it seems that such possibilities are just as close in relevant respects as those where the occurrent outcome statistics reflect the chances. In particular, as the all-heads sequence has some chance of coming to pass, there is (by BCP) a physical possibility sharing history and laws with our world in which all-heads occurs. This looks like a legitimately close possibility to our own.

Prospects look rather better on a reductionist view of chance (Supplement A.3 ). On such a view, we can say that worlds where an infinite sequence of heads does occur at some close possibility will look very different from ours; they differ in law or history to ours. In such worlds, the chance of heads is much closer to 1 (reflecting the fact that if a coin were tossed infinitely many times, it might well land heads on each toss)—the coin is not after all fair. The response, then, is that in any situation where the reductionist chance of heads really is 0.5, suitable outcome sequences in that situation or its nearest neighbours are in fact all unbiased with respect to outcome frequencies. That is to say, they at least satisfy the property of large numbers; and arguably they can be expected to meet other randomness properties also. So, on this view, there is no counterexample to RCT from the mere possibility of these kinds of extreme outcome sequences. This response depends on the success of the reductionist similarity metrics for chancy counterfactuals developed by Lewis (1979a) and Williams (2008); the latter construction, in particular, invokes a close connection between similarity and randomness. (Lewis’ original construction is criticised in Hawthorne 2005.)

However, we needn’t use such an extreme example to make the point. For the same phenomenon exists with almost any unrepresentative outcome sequence. A fair coin, tossed 1000 times, has a positive chance of landing heads more than 700 times. But any outcome sequence of 1000 tosses which contains more than 700 heads will be compressible (long runs of heads are common enough to be exploited by an efficient coding algorithm, and 1000 outcomes is long enough to swamp the constants involved in defining the universal prefix-free Kolmogorov complexity). So any such outcome sequence will not be random, even though it quite easily could come about by chance. The only way to resist this counterexample is to refuse to acknowledge that such a sequence of outcomes can be an appropriate sequence in RCT. This is implausible, for such sequences can be actual, and can be sufficiently long to avoid the analogue of the problem of the single case, certainly long enough for the Kolmogorov definition of randomness to apply. The only reason to reject such sequences as suitable is to save RCT, but that is clearly question begging in this context. In this case of an unrepresentative finite sequence, even reductionism about chance needn’t help, because it might be that other considerations suffice to fix the chance, and so we can have a genuine fair chance but a biased and non-random outcome sequence.

Any given token event is an instance of many different types of trial:

It is obvious that every individual thing or event has an indefinite number of properties or attributes observable in it, and might therefore be considered as belonging to an indefinite number of different classes of things… (Venn 1876: 194)

Suppose Lizzie tosses a coin on Tuesday; this particular coin toss may be considered as a coin toss; a coin toss on a Tuesday; a coin toss by Lizzie; an event caused by Lizzie; etc. Each of these ways of typing the outcome give rise to different outcome sequences, some of which may be random, while others are not. Each of these outcome sequences is unified by a homogenous kind of trial; as such, they may all be suitable sequences to play a role in RCT.

This is not a problem if chance too is relative to a type of trial, for we may simply make the dependence on choice of reference class explicit in both sides of RCT. If chances were relative frequencies, it would be easy enough to see why chance is relative to a type of trial. But chances aren’t frequencies, and single-case chance is almost universally taken to be not only well-defined for a specific event, but unique for that event. We naturally speak of the chance that this coin will lands heads on its next toss, with the chance taken to be a property of the possible outcome directly, and not mediated by some particular description of that outcome as an instance of this or that kind of trial. Moreover, for chance to play its role in the Principal Principle (§ 1 and Supplement A.1 ), there must be a unique chance for a given event that is to guide our rational credence, as we have only one credence in a particular proposition stating the occurrence of that event. (Indeed, the inability of frequentists to single out a unique reference class, the frequency in which is the chance, was taken to be a decisive objection to frequentism.) On the standard understanding of chance, then, there is a mismatch between the left and right sides of the RCT. And this gives rise to a counterexample to RCT, if we take an event with a unique non-trivial single-case chance, but such that at least one way of classifying the trial which produced it is such that the sequence of outcomes of all trials of that kind is not random. The trivial case might be this: a coin is tossed and lands heads. That event happened by chance, yet it is of the type ‘coin toss which lands heads’, and the sequence consisting of all outcomes of that type is not random.

The natural response—and the response most frequentists offered, with the possible exception of von Mises [ 14 ] —was to narrow the available reference classes. (As noted in Supplement A.3 , many frequentists were explicit that chances were frequencies in repetitions of natural kinds of processes.) Salmon (1977) appeals to objectively homogenous reference classes (those which cannot be partitioned by any relevant property into subclasses which differ in attribute frequency to the original reference class). Salmon’s proposal, in effect, is that homogeneous reference classes are random sequences, the evident circularity of which will hardly constitute a reply to the present objection. Reichenbach (1949: 374) proposed to ‘proceed by considering the narrowest class for which reliable statistics can be compiled’, which isn’t circular, but which fails to respond to the objection since it provides no guarantee that there will be only one such class. There may well be multiple classes which are equally ‘narrow’ and for which reliable statistics can be collected (Gillies, 2000: 816). In the present context, this will amount to a number of sequences long enough to make for reliable judgements of their randomness or lack thereof.

This objection requires the chance of an event to be insensitive to reference class. Recently, Hájek (2007) has argued that no adequate conception of probability is immune to a reference class problem, so that this requirement cannot be met. (For a related view of relativised chance, though motivated by quite different considerations, see Glynn 2010.) However, as Hájek notes, this conclusion makes it difficult to see how chance could guide credence, and it remains an open question whether a relativised theory of chance that meets the platitudes concerning chance can be developed.

The two previous problems notwithstanding, many have found the most compelling cases of chance without randomness to be situations in which there is a biased chance process. A sequence of unfair coin tosses will have an unbalanced number of heads and tails, and such a sequence cannot be random. But such a sequence, and any particular outcome in that sequence, happens by chance.

That such sequences aren’t random can be seen by using both Martin-Löf- and Kolmogorov-style considerations. In the latter case, as we have already seen, a biased sequence will be more compressible than an unbiased sequence, if the sequence is long enough, because an efficient coding will exploit the fact that biased sequences will typically have longer subsequences of consecutive digits, and hence will not be random. In the former case, a biased sequence will violate at least one measure one property, on the standard Lebesgue measure on infinite binary sequences—in particular, a measure one subset of the Cantor space will be Borel normal (§ 2.1 ), but no biased sequence is Borel normal. So on the standard account of randomness, no sequences of outcomes of a biased chance process are random, but of course these outcomes happened by chance.

One response to this problem is to try and come up with a characterisation of randomness which will permit the outcomes of biased chances to be random. It is notable that von Mises’ initial characterisation of randomness was expressly constructed with this in mind—for him, a random sequence is one for which there is no admissible subsequence having a frequency differing from the frequency in the original sequence. This account is able to handle any value for the frequency, not only the case where each of two outcomes are equifrequent. Given that the Martin-Löf approach is a generalisation of von Mises’, it is not surprising that it too can be adapted to permit biased sequences to be random. Consider a binary process with outcome probabilities \((p, 1 - p)\). The law of large numbers in a general form tells us that a measure one set of sequences of independent trials of such a process will have limit frequencies of outcomes equal to \((p, 1 - p)\). This measure is not the standard Lebesgue measure, but rather a measure defined by the chance function in question. We can similarly re-interpret the other effective statistical tests of randomness. Drawing as we did above (§ 2.1.2 ) on the language of statistical testing, we can characterise the random sequences as those which are not significant with respect to the hypothesis that the outcome chances are \((p, 1 - p)\)—those which, as it were, conform to our prior expectations based on the underlying chances.

To approach the topic of randomness of biased sequences through Kolmogorov complexity, suppose we are given—somehow—any computable probability measure \(\lambda\) over the set of infinite binary sequences (that is, the probability of a given sequence can be approximated arbitrarily well by a recursive function). A sequence \(\sigma\) is \(\lambda\)-incompressible iff for each \(n\), the Kolmogorov complexity of the length \(n\) initial subsequence of \(\sigma\) (denoted \(\sigma_n )\) is greater than or equal to \({-}\log_2 (\lambda(\sigma_n ))\). Where \(\lambda\) is the Lebesgue measure (supplement B.2 ), it follows that

so we get back the original definition of Kolmogorov complexity in that special case. With this generalised definition of Kolmogorov randomness in hand, it turns out that we can show a generalisation of Schnorr’s theorem (§ 2.3 ): a sequence \(\sigma\) is \(\lambda\)-incompressible iff \(\sigma\) is ML-random with respect to \(\lambda\). [ 15 ] In the framework of supplement B.1.1 , a sequence is ML-random in this generalised sense iff the measure, under our arbitrary computable measure \(\lambda\), of the \(n\)th sequential significance test is no greater than \(1/2^n\). (There are some potential pitfalls, suggesting that perhaps the generalisation to an arbitrary computable measure is an overgeneralisation: see supplement B.1.3 for some details.)

While the above approach, with the modifications suggestion in the supplement taken on board, does permit biased random sequences, it comes at a cost. While the Lebesgue measure is a natural one that is definable on the Cantor space of sequences directly, the generalisation of ML-randomness requires an independent computable probability measure on the space of sequences to be provided. While this may be done in cases where we antecedently know the chances, it is of no use in circumstances where the existence of chance is to be inferred from the existence of a random sequence of outcomes, in line with RCT—for every sequence there is some chance measure according to which it is random, which threatens to trivialise the inference from randomness to chance. As Earman (1986: 145) also emphasises, this approach to randomness seems to require essentially that the chanciness of the process producing the random sequence is conceptually prior to the sequence being random. This kind of process approach to randomness has some intuitive support, and we will return to it below (§ 6.2 ), but it risks turning RCT into an uninformative triviality. By contrast, the Lebesgue measure has the advantage of being intrinsically definable from the symmetries of the Cantor space, a feature other computable measures lack.

The main difficulty with the suggested generalisation to biased sequences lies in the simple fact that biased sequences, while they might reflect the probabilities in the process which produces them, simply don’t seem to be random in the sense of disorderly and incompressible. The generalisation above shows that we can define a notion of disorderliness that is relative to the probabilities underlying the sequence, but that is not intrinsic to the sequence itself independently of whatever measure we are considering. As Earman puts it (in slightly misleading terminology):

[T]here is a concept of randomness and a separable concept of disorder. The concept of disorder is an intrinsic notion; it takes the sequence at face value, caring nothing for genesis, and asks whether the sequence lacks pattern.… By contrast, the concept of randomness is concerned with genesis; it does not take the sequence at face value but asks whether the sequence mirrors the probabilities of the process of which it is a product. There is a connection between this concept of randomness and the concept of disorder, but it is not a tight one. (Earman 1986: 145)

As we might put it: Kolmogorov randomness is conceptually linked to disorderliness, and while we can gerry-rig a notion of ‘biased disorder’, that doesn’t really answer to what we already know about the incompressibility of disorderly sequences. While we might well regard a sequence featuring even numbers of heads and tails, but produced by successive tosses of an unfair coin, to be biased in some sense with respect to the underlying measure of process behind it, it is still plausible that this unrepresentativeness of the sequence isn’t conceptually connected with disorder in any interesting sense. It is very intuitive, as this remark from Dasgupta suggests, to take that biasedness—the increased orderliness of the sequence—to contrast with randomness:

if for a sequence \(x\) this limiting frequency exists but is not equal to 1/2, then, in view of our underlying fair coin model, \(x\) would clearly be biased, not random. … Thus, it is natural to view this ‘stochastic law of unbiasedness’ as a ‘stochastic law of randomness’. (Dasgupta 2011: §3.2)

As the bias in a chance process approaches extremal values, it is very natural to reject the idea that the observed outcomes are random. (As an example, we may consider human behaviour—while people aren’t perfectly predictable, and apparently our behaviour doesn’t obey non-probabilistic psychological laws, nevertheless to say that people act randomly is incorrect.) Moreover, there is a relatively measure-independent notion of disorder or incompressibility of sequences, such that biased sequences really are less disorderly. We can define a measure-dependent notion of disorder for biased sequences only by ignoring the availability of better compression techniques that really do compress biased sequences more than unbiased ones. To generalise the notion of randomness, as proposed above, permits highly non-random sequences to be called random as long as they reflect the chances of highly biased processes. So there is at least some intuitive pull towards the idea that if randomness does bifurcate as Earman suggests, the best deserver of the name is Kolmogorov randomness in its original sense. But this will be a sense that contrasts with the natural generalisation of ML-randomness to deal with arbitrary computable probability measures, and similarly contrasts with the original sense of randomness that von Mises must have been invoking in his earliest discussions of randomness in the foundations of probability.

In light of the above discussion, while there has been progress on defining randomness for biased sequences in a general and theoretically robust way, there remain difficulties in using that notion in defence of any non-trivial version of RCT, and difficulties in general with the idea that biased sequences can be genuinely disorderly. But the generalisation invoked here does give some succour to von Mises, for a robust notion of randomness for biased sequences is a key ingredient of his form of frequentism.

A further counterexample to RCT, related to the immediately previous one, is that randomness is indifferent to history, while chance is not. Chance is history-dependent. The simplest way in which chance is history-dependent is when the conditions that may produce a certain event change over time:

Suppose you enter a labyrinth at 11:00 a.m., planning to choose your turn whenever you come to a branch point by tossing a coin. When you enter at 11:00, you may have a 42% chance of reaching the center by noon. But in the first half hour you may stray into a region from which it is hard to reach the center, so that by 11:30 your chance of reaching the center by noon has fallen to 26%. But then you turn lucky; by 11:45 you are not far from the center and your chance of reaching it by noon is 78%. At 11:49 you reach the center; then and forevermore your chance of reaching it by noon is 100%. (Lewis, 1980: 91)

But there are more complicated types of history-dependence. In Lewis’ example, the property which changes to alter the chances is how close the agent is to the centre. But there are cases where the property which changes is a previous outcome of the very same process. Indeed, any process in which successive outcomes of repeated trials are not probabilistically independent will have this feature.

One example of chance without randomness involves an unbiased urn where balls are drawn without replacement. Each draw (with the exception of the last) is an event which happens by chance, but the sequence of outcomes will not be random (because the first half of the sequence will carry significant information about the composition of the second half, which may aid compressibility). But a more compelling example is found in stochastic processes in which the chances of future outcomes depend on past outcomes. One well-known class of such process are Markov chains, which produce a discrete sequence of outcomes with the property that the value of an outcome is dependent on the value of the immediately prior outcome (but that immediately prior outcome screens off the rest of the history). A binary Markov chain might be the weather (Gates and Tong 1976): if the two possible outcomes are ‘sunny’ and ‘rainy’, it is plausible to suppose that whether tomorrow is rainy depends on whether today was rainy (a rainy day is more likely to be preceded by another rainy day); but knowing that today was rainy arguably makes yesterday’s weather irrelevant.

If a Markov chain is the correct model of a process, then even when the individual trial outcomes happen by chance, we should expect the entire sequence of repeated trials to be non-random. In the weather case just discussed, we should expect a sunny day to be followed by a sunny day, and a rainy day by a rainy one. In our notation 11 and 00 should be more frequent than 10 or 01. But the condition of Borel normality, which all random sequences obey, entails that every finite sequence of outcomes of equal length should have equal frequency in the sequence. So no Borel normal sequence, and hence no random sequence, can model the sequence of outcomes of a Markov chain, even though each outcome happens by chance.

At least some non-random sequences satisfy many of the measure one properties required of random sequences. For example, the Champernowne sequence, consisting of all the binary numerals for every non-negative integer listed consecutively (i.e., 011011100101110111…), is Borel normal. This sequence isn’t random, as initial subsequences of reasonable length are highly compressible. But it looks like it satisfies at least some desiderata for random sequences. This sequence is an attempt at producing a pseudorandom sequence—one that passes at least some statistical tests for randomness, yet can be easily produced. (The main impetus behind the development of pseudorandom number generators has been the need to efficiently produce numbers which are random for all practical purposes, for use in cryptography or statistical sampling.) Much better examples exist than the Champernowne sequence, which meet more stringent randomness properties. [ 16 ] One simple technique for generating pseudorandom sequences is a symbol shift algorithm (Smith 1998: 53). Given an initial ‘seed’ numeral \(s_1, s_2 , \ldots ,s_n\), the algorithm simply spits out the digits in order. Obviously this is useless if the seed is known, or can in some way be expected to be correlated with the events to which one is applying these pseudorandom numbers. But in practical applications, the seed is often chosen in a way that we do expect it to carry no information about the application (in simple computer pseudorandom number generators, the seed may be derived in some way from the time at which the seed is called for). With a finite seed, this sequence will obviously repeat after a some period. A symbol shift is the simplest possible function from seed to outcome sequence; better algorithms use a more complicated but still efficiently computable function of the seed to generate outcome sequences with a longer period, much longer than the length of the seed (e.g., the ‘Mersenne twister’ of Matsumoto and Nishimura 1998 has a period of \(2^{19937} - 1)\).

If the seed is not fixed, but is chosen by chance, we can have chance without randomness. For example, suppose the computer has a clock representing the external time; the time at which the algorithm is started may be used as a seed. But if it is a matter of chance when the algorithm is started, as it well may be in many cases, then the particular sequence that is produced by the efficient pseudorandom sequence generator algorithm will be have come about by chance, but not be random (since there is a program which runs the same algorithm on an explicitly given seed; since the seed is finite, there will be such a program; and since the algorithm is efficient, the length before the sequence produced repeats will be longer than the code of the program plus the length of the seed, making the produced sequence compressible). Whether the seed is produced by chance or explicitly represented in the algorithm, the sequence of outcomes will be the same—one more way in which it seems that the chanciness of a sequence can vary while whether or not it is random remains constant. (Symbol shift dynamics also permit counterexamples to the other direction of RCT—see § 5.2 .)

Much the same point could have been made, of course, with reference to any algorithm which may be fed an input chosen by chance, and so may produce an outcome by chance, but where the output is highly compressible. (One way in which pseudorandom sequence generators are nice in this respect is that they are designed to produce highly compressible sequences, though non-obviously highly compressible ones). The other interesting thing about those algorithms which produce pseudorandom sequences is that they provide another kind of counterexample to the epistemic connection between chance and randomness. For our justification in thinking that a given sequence is random will be based on its passing only finitely many tests; we can be justified in believing pseudorandom sequence to be random (in some respectable sense of justification, as long as justification is weaker than truth), and justified in making the inference to chance via RCT. But then we might think that this poses a problem for RCT to play the right role epistemically, even if it were true. Suppose one sees a genuinely random sequence and forms the justified belief that it is random. The existence of pseudorandom sequences entails that things might seem justificatorily exactly as they are and yet the sequence not be random. But such a scenario, arguably, defeats my knowing that the sequence is random, and thus defeats my knowing the sequence to have been produced by chance (and presumably undermines the goodness of the inference from randomness to chance).

5. Randomness Without Chance

The counterexamples to RCT offered in §§ 4.3 – 4.4 suggest strongly that the appeal of RCT depends on our curious tendency to take independent identically distributed trials, like the Bernoulli process of fair coin tossing, to be paradigms of chance processes. Yet when we widen our gaze to encompass a fuller range of chance processes, the appeal of the right-to-left direction of RCT is quite diminished. It is now time to examine potential counterexamples to the other direction of RCT. There are a number of plausible cases where a random sequence potentially exists without chance. Many of these cases involve interesting features of classical physics, which is apparently not chancy, and yet which gives rise to a range of apparently random phenomena. Unfortunately some engagement with the details of the physics is unavoidable in the following.

One obvious potential counterexample involves coin tossing. Some have maintained that coin tossing is a deterministic process, and as such entirely without chances, and yet which produces outcome sequences we have been taking as paradigm of random sequences. This will be set aside until § 7 , where the claim that determinism precludes chance will be examined.

For many short sequences, even the most efficient prefix-free code will be no shorter than the original sequence (as prefix-free codes contain information about the length of the sequence as well as its content, if the sequence is very short the most efficient code may well be the sequence itself prefixed with its length, which will be longer than the sequence). So all short sequences will be Kolmogorov random. This might seem counterintuitive, but if randomness indicates a lack of pattern or repetition, then sequences which are too short to display pattern or repetition must be random. Of course it will not usually be useful to say that such sequences are random, mostly because in very short sequences we are unlikely to talk of the sequence at all, as opposed to talking directly about its constituent outcomes.

Because of the modal aspect of RCT, for most processes there will possibly be a long enough sequence of outcomes to overcome any ‘accidental’ randomness due to actual brevity of the outcome sequence. But for events which are unrepeatable or seldom repeatable, even the merely possible suitable reference classes will be small. And such unrepeatable events do exist—consider the Big Bang which began our universe, or your birth ( your birth, not the birth of qualitatively indistinguishable counterpart), or the death of Ned Kelly. These events are all part of outcome sequences that are necessarily short, and hence these events are part of Kolmogorov random sequences. But it is implausible to say that all of these events happened by chance; no probabilistic theory need be invoked to predict or explain any of them. In the case of Kelly’s death, for example, while it may have been a matter of chance when he happened to die, it was not chance that he died, as it is (physically) necessary that he did. So there are random sequences—those which are essentially short—in which each outcome did not happen by chance.

The natural response is to reject the idea that short sequences are apt to be random. The right hand side of RCT makes room for this, for we may simply insist that unrepeatable events cannot be repeated often enough to give rise to an adequate sequence (whether or not the inadequate sequence they do in fact give rise to is random). The problem here is that we can now have chance without randomness, if there is a single-case unrepeatable chance event. Difficulties in fact seem unavoidable. If we consider the outcomes alone, either all short sequences are random or none of them are; there is no way to differentiate on the basis of any product-based notion between different short sequences. But as some single unrepeatable events are chancy, and some are not, whichever way we opt to go with respect to randomness of the singleton sequences of such events we will discover counterexamples to one direction or another of RCT.

The simple symbol shift dynamics in § 4.5 had a finite seed, which allowed for chance without randomness. Yet there seem to be physical situations in which a symbol shift dynamics is an accurate way of representing the physical processes at work. One simple example might be a stretch and fold dynamics, of the kind common in chaos theory (Smith 1998: §4.2). The classic example is the baker’s transformation (Earman 1986: 167–8; Ekeland 1988: 50–9). We take a system the state of which at any one time is characterised by a point \((p, q)\) in the real unit square. We specify the evolution of this system over time as follows, letting \(\phi\) be the function governing the discrete evolution of the system over time (i.e., \(s_{t + 1} = \phi(s_t))\):

This corresponds to transforming the unit square to a rectangle twice as wide and half the height, chopping off the right half, and stacking it back on top to fill the unit square again. (That this transformation reminded mathematicians of baking says something about their unfamiliarity with the kitchen—similar transformations, where the right half is ‘folded’ back on top, are more realistic.) If we represent the coordinates \(p\) and \(q\) in binary, the transformation is this:

So this is a slight variant on a simple symbol shift, as the \(p\)-coordinate is a symbol shift to the right, while the \(q\) coordinate is, in effect, a symbol shift to the left. [ 17 ]

One important feature of this dynamics is that it is measure preserving, so that if \(X\) is a subset of the unit square, \(\mu(X) = \mu(\phi(X))\). (This is easily seen, as the symbol shift dynamics on the basic sets of infinite binary sequences is measure-preserving, and each coordinate can be represented as an infinite binary sequence.) Define the set \(L = \{(p, q) : 0 \le p \lt \frac{1}{2} \}\). We see that \((p, q) \in L\) iff \(p_1 = 0\). Since \(p\) can be represented by an infinite binary sequence, and a measure one set of finite binary sequences is Borel normal, we see that almost all states of this system are such that, over time, \(\mu(\phi(s) \in L \mid s \in L) = \mu(\phi(s) \in L)\)—that is, whether the system is in \(L\) at \(t\) is probabilistically independent of its past history. Furthermore, \(\mu(L) = \mu(\overline{L})\). The behaviour of this system over time, with respect to the partition \(\{L, \overline{L}\}\), is therefore a Bernoulli process, exactly like a sequence of fair coin tosses—a series of independent and identically distributed repetitions of a chance process. If the RCT is true, then a system which behaves exactly like a chance process should have as its product a random sequence. So the sequence of outcomes for the baker’s transformation (under this partition) is a random sequence.

But unlike a sequence of fair coin tosses, supposing them to involve genuine chance, the baker’s transformation is entirely deterministic . Given a particular point \((p, q)\) as an initial condition, the future evolution of states of the system at each moment is time \(t\) is determined to be \(\phi^t (p, q)\). So while the product produced is random, as random as a genuine chance process, these outcomes do not happen by chance; given the prior state of the system, the future evolution is not at all a matter of chance. So we have a random sequence without chance outputs. (Indeed, given the symbol shift dynamics, the evolution of the system over time in \(\{L, \overline{L}\}\) simply recapitulates successive digits of the starting point.) To be perfectly precise, the trial in this case is sampling the system at a given time point, and seeing which cell of the coarse grained partition it is in at each time. This is a sequence of arbitrarily repeated trials which produces a random sequence; yet none of these outcomes happens by chance. [ 18 ] To put the point slightly differently: while the sequence of outcomes is random, there is a perfectly adequate theory of the system in question in which probability plays no role. And if probability plays no role, it is very difficult to see how chance could play a role, since there is no probability function which serves as norm for credences, governs possibility, or is non-trivial and shared between intrinsic duplicate trials. In short, no probability function that has the features required of chance plays a role in the dynamics of this system, and that seems strong reason for thinking there is no chance in this system.

The baker’s transformation provides a simple model of deterministic macro-randomness —a system which has a \(\mu\)-measure-preserving temporal evolution, and produces a sequence of coarse grained states that have the Bernoulli property. It is a question of considerable interest whether there are physically more realistic systems which exhibit the same features. We may conceive of an \(n\)-particle classical (Newtonian) system as having, at each moment, a state that is characterised by a single point in a \(6n\)-dimensional state space (each point a \(6n\)-tuple characterising the position and momentum of each particle). The evolution of the system over time is characterised by its Hamiltonian , a representation of the energetic and other properties of the system. The evolution under the Hamiltonian is \(\mu\)-measure-preserving (by Liouville’s theorem), so it might be hoped that at least some systems could be shown to be Bernoulli too. Yet for closed systems, in which energy is conserved over time, this is not generally possible. Indeed, for closed systems it is not generally possible to satisfy even a very weak randomness property, ergodicity . A system is ergodic just in case, in the limit, with probability one, the amount of time a system spends in a given state is equal to the (standard) measure of state space that corresponds to that state (Earman 1986: 159–61; Sklar 1993: 60–3; Albert 2000: 59–60). While a Bernoulli system is ergodic, the converse entailment does not hold; if the system moves only slowly from state to state, it may be ergodic while the state at one time is strongly dependent on past history (Sklar 1993: 166–7). While ergodicity has been shown to hold of at least one physically interesting system (Yakov Sinai has shown that the motion of hard spheres in a box is ergodic, a result of great significance for the statistical mechanics of ideal gases), a great many physically interesting systems cannot be ergodic. This is the upshot of the so-called KAM theorem, which says that for almost all closed systems in which there are interactions between the constituent particles, there will be stable subregions of the state space—regions of positive measure such that if a system is started in such a region, it will always stay in such a region (Sklar 1993: 169–94). Such systems obviously cannot be ergodic.

The upshot of this for our discussion may be stated: ‘there are no physically realistic classical systems which exhibit even ergodicity, and so no physically realistic classical system can exhibit randomness. The baker’s transformation is a mathematical curiosity, but is not a genuine case of randomness without chance, since systems like it are not physically possible.’ This response is premature. There are physically interesting systems to which the KAM theorem does not apply. Open or dissipative systems, those which are not confined to a state space region of constant energy, are one much studied class, because such systems are paradigms of chaotic systems. The hallmarks of a chaotic dissipative system are two (Smith 1998: §1.5):

• There exists at least one set of states in state space \(A\) (an attractor ) such that when the system is started initially in its neighbourhood \(N(A)\), the trajectory of the system will in the limit end up in \(A\) (\(\lim_{t \rightarrow \infty} \phi^{t}N(A) \subseteq A)\); and
• The system displays sensitive dependence on initial conditions: that is, in some set of state space points that are all within some arbitrary distance \(\delta\) of each other, there are at least two points whose subsequent trajectories diverge by at least \(\varepsilon\) after some time \(t\). [ 19 ]

There are physically realistic classical systems which exhibit both of these features, of which the best known is perhaps Lorenz’s model of atmospheric convection (Smith 1998: §1.4; Earman 1986: 165). The combination of these two features permits very interesting behaviour—while the existence of an attractor means that over time the states of the system will converge to the region of the attractor, the sensitive dependence on initial conditions means that close states, at any time, will end up arbitrarily far apart. For this to happen the attractor must have a very complex shape (it will be a region of small measure, but most of the state space will be in the neighbourhood of the attractor). More importantly for our purposes, a system with these characteristics, supposing that the divergence under evolution of close states happens quickly enough, will yield behaviour close to Bernoulli—it will yield rapid mixing (Luzzatto et al. 2005). [ 20 ] Roughly, a system is mixing iff the presence of the system in coarse state at one time is probabilistically independent of its presence in another coarse state at another time, provided there is a sufficient amount of time between the two times . This is weaker than Bernoulli (since the states of a Bernoulli system are probabilistically independent if there is any time between them), but still strong enough to plausibly yield a random sequence of outcomes from a coarse grained partition of the state space, sampled infrequently enough. So we do seem to have physically realistic systems that yield random behaviour without chance. (See also the systems discussed in Frigg 2004.)

Indeed, the behaviour of a chaotic system will be intuitively random in other ways too. The sensitive dependence on initial conditions means that, no matter how accurate our finite discrimination of the initial state of a given chaotic system is, there will exist states indiscriminable from the initial state (and so consistent with our knowledge of the initial state), but which would diverge arbitrarily far from the actual evolution of the system. No matter, then, how well we know the initial condition (as long as we do not have infinite powers of discrimination), there is another state the system could be in for all we know that will evolve to a discriminably different future condition. Since this divergence happens relatively quickly, the system is unable to be predicted. (Anecdotally, at least, Lorenz’ model of the weather seems borne out by our inability to reliably predict future weather even a few days from now.) Insofar as randomness and lack of reliable prediction go hand in hand, we have another reason for thinking there is product randomness here (§ 6.2 ).

Just as before, the classical physical theory underlying the dynamics of these chaotic systems is one in which probability does not feature. So we are able to give an adequate characterisation of the physical situation without appeal to any probabilities fit to play the chance role. Given well-enough behaved boundary conditions, this system is also deterministic (though see § 5.3 ), and that may also be thought to preclude a role for non-trivial chances. So again we have randomness in the performance, though none of the outcomes happened by chance.

Two avenues of resistance to the line of thought in this section suggest themselves. The first is to maintain that, despite the truth of determinism for the baker’s transformation and classical physics (modulo worries to be addressed in the following section), there are still, or at there least may be, non-trivial chances in these theories. The proposal is that the possibility remains of deterministic chance , so that from the fact of determinism alone it does not follow that we have a counterexample to RCT. The outcomes may be determined to happen by the prior conditions, but (so the suggestion goes) they may still happen by chance. This radical proposal is discussed below, in § 7 . It should be noted, however, that even if it is true that deterministic chance is possible, that observation is far from establishing that the physical theories under discussion here are ones in which deterministic chance features. That some deterministic theories may have chances is no argument that all will, and particularly in very simple cases like the baker’s transformation, there doesn’t seem much point in an invocation of deterministic chance: chances will be trivial or redundant if classical physics is true. The second avenue of resistance is to claim that there really is chance here—it is chance of the initial conditions which is recapitulated in the random sequence of outcomes. While the Lebesgue measure over the set of initial conditions in our models is formally like a probability function, to assume it yields genuine chances is a fairly controversial thesis (for a contrary opinion, see Clark 1987). Other initial conditions could have obtained; still it seems wrong to think that (somehow) there was a chance process which terminated in the selection of the initial conditions which happened in fact to obtain at our world. Rather, that the initial conditions are what they are seems to be a brute fact. If there are to be chances, then, they cannot be dynamical chances, the kind that is familiar from physics, and from our discussion in § 1.2 . Some recent arguments in favour of the possibility of chancy initial conditions are discussed in the following supplementary document:

Supplement D. Chance and Initial Conditions

But whether or not the idea of chancy initial conditions can be made to work, the fact remains that at most one outcome in the random sequence—the first one—happens by chance. The subsequent states do not, yet RCT is committed to there being (dynamical, transition) chances in those state transitions.

Despite the neat picture of classical determinism drawn in the previous section, it is well known that classical physics is not in fact deterministic. These cases of indeterminism do not undermine the applications of classical mechanics in the previous section. But classical indeterminism may provide problems of its own for RCT. Useful further material on the topic of this section can be found in the entry on causal determinism ( Hoefer 2010 : §4.1 ).

For present purposes, indeterminism occurs when the state of the system at one time does not uniquely fix the state the system will be in at some future time (see § 7 ). To show indeterminism in the classical case, it suffices to give a state of some system at a given time and to specify two future states that are incompatible with each other and yet both states are consistent with Newton’s laws of motion and the initial state.

To help us in this task, it is useful to note one fact about Newtonian mechanics: the laws are time-reversal invariant (Albert 2000: ch. 1). That is, for every lawful trajectory through the state space, there is another lawful trajectory which results from the first by mapping every instantaneous state in the first trajectory to its image state in which the particles have the same positions but the signs on the components of momentum are reversed, and running the trajectory in reverse order. These image states are those where the particles are in the same positions but moving in exactly the opposite direction. So for every lawful process, that process running backwards is also lawful. (If these trajectories are so lawful, why don’t we see them?—that is the deep question of thermodynamic asymmetry, discussed briefly in supplement D .)

Two examples serve to illustrate the possibility of classical indeterminism. A very elegant recent example is provided by ‘Norton’s dome’ (Norton 2003; 2008). A point mass is at rest at the apex of a dome (of certain shape) at \(t^*\). One obvious solution to Newton’s equations of motion is that it continues to do so for all moments \(t \gt t^*\). But, Norton points out, there is another solution: that while at \(t = t^*\) the mass is at rest, at every moment \(t \gt t^*\), the mass is moving in some direction. But this means the mass spontaneously moves off in some arbitrary direction at an arbitrary time. [ 21 ] Determinism is clearly violated: for some given time \(t^*\), there is a state at \(t'\) where the particle remains at the apex of the dome; and there are many incompatible states where at \(t'\) the particle is somewhere else on the surface of the dome. Nothing in the conditions at \(t^*\) fixes which of these many future states will be the one to eventuate. An easy way to understand the dome example is to consider its time reversal: a ball is given some initial velocity along the surface of the dome toward the apex. Too little, and it falls short; too great, and it overshoots. Just right, however, and the ball comes to rest precisely at the apex, and remains at rest. The time reversal of this system is the original dome example.

A more exotic example involves ‘space invaders’ (Earman 1986: ch. III). These are particles that are at no spatial location at time \(t\), and therefore form no part of the state at time \(t\), but which travel in ‘from’ spatial infinity and by time \(t'\) have a location. We can see the example more clearly if we invoke time-reversal invariance. Consider two point particles, \(a\) and \(b\), at rest and in the vicinity of one another at \(t^*\). From \(t^*\) onwards, force is applied to \(a\) in such a way that the velocity of \(a\) away from \(b\) increases without bound. This is possible, because there is no upper bound on velocity in classical physics. Indeed, let the velocity of \(a\) increase fast enough that by some finite time \(t', a\) has no finite velocity, and is thus ‘at’ spatial infinity. The time-reversal of this system has the particle \(a\) having no location at \(t'\), but having a location and continuously decreasing velocity at every moment \(t \gt t'\), until it comes to rest at \(t^*\). This system violates determinism in the sense given above. The state at \(t'\) consists of a single particle \(b\) at rest. That state could be followed at \(t^*\) by the space-invaded state just described; or it could be followed at \(t^*\) by the incompatible state of \(b\) simply being at rest some more. Nothing in the laws rules out either of these transitions. Of course, the model isn’t particularly physically realistic—where does the force acting on \(a\) come from? But physically more realistic systems exhibiting the same general structure have been given; Earman mentions one due to Mather and McGehee (1975) involving four point particles moving in such a way that the forces they exert on each other in collisions end up unboundedly far from one another in a finite time (see also Saari and Xia 1995).

While classical mechanics is thus indeterministic, it is importantly not chancy. There is no reason to think that we either need to or can assign a probability distribution over the possible future states in our indeterministic systems. Norton says this about his dome:

One might think that … we can assign probabilities to the various possible outcomes. Nothing in the Newtonian physics requires us to assign the probabilities, but we might choose to try to add them for our own conceptual comfort. It can be done as far as the direction of the spontaneous motion is concerned. The symmetry of the surface about the apex makes it quite natural for us to add a probability distribution that assigns equal probability to all directions. The complication is that there is no comparable way for us to assign probabilities for the time of the spontaneous excitation that respect the physical symmetries of solutions. Those solutions treat all candidate excitation times \(T\) equally. A probability distribution that tries to make each candidate time equally likely cannot be proper—that is, it cannot assign unit probability to the union of all disjoint outcomes. [ 22 ] Or one that is proper can only be defined by inventing extra physical properties, not given by the physical description of the dome and mass, Newton’s laws and the laws of gravitation, and grafting them unnaturally onto the physical system. (Norton 2003: 9–10)

The point Norton makes about the impossibility of a uniform distribution over a countable set of time intervals holds also for the time at which we might expect the space invader to occur in the second type of case. Thus, it seems, we have indeterminism without chance.

We can use these constructions to come up with counterexamples to RCT. Let a dome system be prepared, and left for a period of 5 seconds. If the ball remains at rest on the apex, call the outcome ‘0’. If the ball moves, and so is at the end of 5 seconds somewhere on the dome other than the apex, call the outcome ‘1’. Both outcomes are physically possible final states of the system after 5 seconds. If the system is repeatedly prepared in this state, it is physically possible to get a random sequence of outcomes of these repeated trials. Certainly the outcomes cannot be produced by some finite algorithm, since the indeterministic dynamics permits as physically possible every sequence of outcomes, including those that differ at some place from every algorithmically compressible sequence. In the infinite future case, it is physically possible for the system to produce every infinite binary sequence, but at most countably many of these are non-random. So it is physically possible for these setups to produce a random sequence of outcomes in the KML sense. But we then have randomness without a chance distribution over the outcomes. Randomness only requires two distinguishable possible outcomes and the possibility of production of arbitrary sequences of such outcomes. Chance requires two distinguishable outcomes, each of which has some chance. These cases show that chance and possibility come apart—there are cases where there are two possible outcomes of a process, neither of which has any chance at all (not even a chance of zero).

The last objection draws on a remark made in § 4.4 . Consider a large finite urn full of black and white balls. The number of balls is sufficiently large that the sequence of outcomes of draws is long enough to be random. So suppose we make a random selection from this urn, drawing balls without replacement until the urn is empty. The resulting sequence of outcomes is, or at least could be, random—it had better be, since this sequence meets all the conditions to be a simple random sample from the population. (We could attach a number to each member of an experimental population, and give the \(n\)-th member a pill of an active substance (respectively, a placebo) if the \(n\)-th draw is black (white).) But in this process, the outcomes become less and less chancy as the number of balls diminishes. Since there are only finitely many balls, there will a draw such that the chance of it being black is either 1 or 0, and so whatever outcome eventuates did not occur by chance. But then we have a random sequence that includes an outcome (drawing a black ball, say) which did not happen by chance, contrary to RCT.

One response is to say that this last outcome did happen by chance, since at the start of the drawing there was a positive chance that a white ball would be drawn last, and a positive chance that a black ball would be. This response neglects the time-dependence of chances. If there were \(n\) balls initially, and we let ‘Lastie’ name the ball that was in actual fact drawn last, then we can say: the chance that Lastie is drawn last was \(1/n\) initially, and after the \(m\)-th draw it was \(1/(n - m)\), until it reached 1 and remained there. At that last stage it was no longer a matter of chance whether Lastie would be drawn last; it is the only ball left in the urn. RCT maintains that a given outcome happens by chance iff it is part of a random sequence. At every time, the event of Lastie being drawn last is part of a random sequence. But then there is at least one time at which Lastie being drawn last is part of random sequence but, at that time, it did not happen by chance. (Alternatively, of course, it could be maintained that even events with trivial chances happen by chance. But this would permit again the problem of biased sequences; a string of all heads tossed by a two-headed coin could then be random, which it is not.)

6. Saving the Thesis: Alternative Conceptions of Chance and Randomness

The discussion in §§ 4 – 5 leaves RCT in a doubtful position. But it may be that the problems for RCT are due more to some defect in the theories of chance and randomness sketched in §§ 1 – 2 . As noted earlier, there are alternative conceptions of chance and randomness that have some appeal and might perhaps save RCT. They won’t have much to say about the modal problems for merely possible random sequences mentioned when RCT was first introduced. But perhaps the other objections can be avoided. The problems for RCT arise fundamentally because of the split between product randomness and process chance. Closing this gap promises to aid RCT. This following two subsections will consider product conceptions of chance and process based conceptions of randomness.

The frequency theory is a product conception of chance. An outcome-type has a chance, according to von Mises (1957), just in case it is part of a random sequence of outcomes in which that outcome type has a robust limit relative frequency. So chance can’t be separated from randomness; it in fact requires randomness. Moreover, since having a limit relative frequency of \(\frac{1}{2}\) is a measure one property of infinite binary sequences, all random sequences define collectives. (Those infinite binary sequences which do not converge on a limit frequency are non-random.) Yet the problems with frequentism as a theory of chance are well known (Hájek, 1997; 2009; Jeffrey, 1977)—we have come across some of them above—and to save RCT at the price of accepting frequentism has not attracted many.

But the prospects are more promising for Humean views of chance (Lewis 1994; Loewer 2004), like the reductionist views discussed in supplements A.1 , A.3 , and D . These are a species of product conception of chance, for a possible world will not feature chances unless the best (simplest, best fitting, and most informative) description of what events occur in that world involves a probability function. Two worlds couldn’t differ in their chances without differing also somewhere in the events which occur. Chance thus supervenes on the actual outcomes in a given world, but not necessarily in a direct way—in the service of simplicity, some deviation from the actual frequency values may yield a description that is better overall. There is considerable debate over whether a Humean best systems account can explain all of the things we know about chance. Lewis thought that chance was the ‘big bad bug’ for his broadly Humean world view (though he thought that the NP, discussed in supplement A.1 , debugged Humean Supervenience: Lewis 1994), and there has been considerable debate over whether or not the PP or the BCP can be accounted for by the Humean, as the references in the supplement A attest. Moreover, there are problems about whether the notion of a probability distribution ‘fitting’ a set of outcomes makes sense (Elga 2004). But suppose a best systems account can be made to work.

The role of simplicity in Humean views is important for randomness. For if a world contains a genuinely random sequence of outcomes, there will be no short description of that sequence. Those descriptions which do not attempt to describe what happens in all its particularity, but instead involve a probability distribution that makes the sequence a typical one with respect to that probability function, will be less informative but much shorter, and still fit just as well. So it seems that if a world contains a random sequence of outcomes, the best theory will be one which involves probabilities, and in virtue of being part of the best theory, those probabilities will be chances. That makes the right to left direction of RCT plausible. The other direction would hold if this route through simplicity was the only way in which chances could enter into the best system. Could there be a world with chances in which there was no random sequence? Here things are rather murkier for the Humean. For the hallmark of the best systems approach to chance is that, by involving simplicity, it avoids some of the problems for pure frequentism. In particular, an unrepresentative outcome sequence (a short one, or highly biased one) need not force the chances to take the value of the actual frequencies. Suppose a world contains two intrinsic duplicate coins, one of which is tossed many times, landing heads about half the time; the other is tossed few times and lands tails every time. The second coin’s outcome sequence has a heads frequency of zero. It is a strength of the best systems account that we may still say the chance of heads was \(\frac{1}{2}\), because the coin is embedded in a world where another, similar, coin did have the appropriate outcome frequencies, and it is overall a simpler theory to say that both of these coins obeyed the same probabilistic laws than that the second one was an oasis of determinism. But this case provides a problem for RCT—it looks like the second coin toss is not part of any random sequence of outcomes (since a few all tails tosses is not random), but it has a chance.

We should not let even the partial success of best systems analysis in preserving RCT sway us. The Humean account of chance is perfectly compatible with the existence of bias and with non-independent sequences of trials; RCT is not. The problem just mentioned arises even in the best circumstances for RCT, where there is at least one actual unbiased fair coin sequence. The existence of such a problem could have been predicted. The best systems account deviates from pure frequentism precisely in trying to accommodate the single-case process intuitions that, as we saw in § 1 , characterise chance. Every success in this direction brings this broadly product conception of chance closer to a process conception, and will therefore be a potential opportunity for counterexamples to RCT to emerge.

As mentioned at the beginning of § 2 , sometimes ‘random’ is used in a process sense. There have been some philosophical approaches to randomness which attempt to take this seriously, but which do not take it to be merely equivalent to ‘chancy’ and thus trivialise RCT. The most popular such approach is to connect randomness with indeterminism , and to defend RCT by arguing directly that indeterminism yields both chance and randomness. Prospects for that approach will be discussed in § 7 .

The next most discussed view of process randomness is an epistemic one. Random processes on this view are those whose outcomes we cannot know in advance; that is, random processes are unpredictable (Eagle 2005; Kyburg 1974: ch. 9). [ 23 ] Here is one clear expression of the view:

At the most basic level, we say an event is random if there is no way to predict its occurrence with certainty. Likewise, a random process is one for which we are not able to predict what happens next. (Frigg 2004: 430)

For this kind of view to yield the right results, we cannot count as ‘being able to predict a process’ if we merely guess rightly about its outcomes. For that reason, prediction must involve some notion of reasonableness ; it must be rational for the agent to make the predictions they do. For example, Eagle (2005: 770) insists that it is the physical theory accepted by the predicting agent that makes certain posterior credences reasonable; simply guessing will not be reasonable, even if it’s correct.

This definition overlaps considerably with those definitions of randomness canvassed in § 2 . In particular, if a process is predictable, that will make available a winning betting strategy on the sequence of outcomes of that process, which cannot therefore be a KML-random sequence of outcomes. So unpredictability of the generating process is a necessary condition on KML-randomness of any concrete outcome sequence.

But unpredictability is not sufficient, for it may be that we cannot know everything true of the future outcomes of the process, and those truths might preclude a KML-random sequence. One way to see this draws on our discussion of chaotic dynamics. Let’s say that a system exhibits apparent dependence on initial conditions if states indiscriminable to us may end up arbitrarily far apart after some relatively short period of time. (Another definition, if knowledge obeys a margin for error principle, would be: a system exhibits apparent dependence on initial conditions if, for all we know, it is sensitively dependent on initial conditions.) Apparent dependence on initial conditions can obtain even if sensitive dependence on initial conditions does not. For there may well be a size \(v\) such that, under some partition of the set of states into regions of measure smaller than \(v\), any two points starting in the same cell of the partition evolve to future states which are close to one another—as long as \(v\) is small with respect to our abilities to discriminate. A system which is apparently but not sensitively dependent on initial conditions will be unpredictable to us, but there will exist an algorithm that, given some finite specification of the initial conditions, generates the future evolution of the system perfectly. (The key is that the finite specification must involve more precise data than we could know to characterise the system.) This sequence, if long enough, is not KML-random, but it is unpredictable. [ 24 ]

Because of the considerable overlap between the unpredictably generated sequences and the KML-random sequences (Frigg 2004: 431), many roles the latter can play will be played by the former too. Eagle (2005) further argues that the unpredictably generated sequences are a better fit to the theoretical role of randomness, and claims on that basis that randomness is unpredictability . One nice thing about this thesis is that, being a process notion, it directly connects with chance processes. We can thus directly evaluate the original thesis connecting chance and randomness, CT, in the form:

(CTU) A process is chancy iff it is not rationally predictable.

The left-to-right direction of CTU looks relatively secure when we attend just to independent and identically distributed trials. But when the trials are not independent, like the examples in § 4.4 , future outcomes can happen by chance, even if knowing the past states of the system does put one in a position to better predict the future outcomes. The right-to-left direction of CTU is in even worse straits. For if KML-randomness of a sequence entails its unpredictable generation, then every counterexample to RCT which involves KML-randomness without chance will also involve unpredictability without chance, and also constitute a counterexample to CTU. There is no succour to be found for defenders of RCT in this conception of randomness.

7. Chance, Randomness, and Determinism

One last hope for the thesis that chancy outcomes are random comes from the connection of both notions with indeterminism. Consider this argument:

\(\mathbf{P1}\): An outcome happens by chance iff it is produced by an indeterministic process. \(\mathbf{P2}\): A possible sequence of outcomes is random iff a repeated indeterministic process could produce all of the outcomes. \(\mathbf{RCT}\): Therefore, an outcome happens by chance iff there is a possible random sequence of outcomes, produced by the same process, of which it is a member.

This argument is valid. If the premises are true, we have a direct argument for RCT. We have no direct response to the objections raised in §§ 4 – 5 , but somehow, if this argument succeeds, those objections must miss the point. The premises have some initial plausibility (though P2 is dubious: surely a properly indeterministic process could produce any sequence of outcomes whatsoever, including many non-random sequences? We discuss this further in §7.2). The thesis that indeterminism is necessary and sufficient for chance has long been a popular claim. And randomness and indeterminism also seem to have a close connection. But to evaluate them, we will need to be more precise than we have been about determinism.

Earman-Montague Determinism : A scientific theory is deterministic iff any two sequences of states in models of that theory which share some state at some time share every state at every time. A theory is indeterministic iff it is not deterministic; equivalently, if two systems can be in the same state at one time and evolve into distinct states. A system is (in)deterministic iff the theory which completely and correctly describes it (of which it is a model) is (in)deterministic. (Earman 1986; Montague 1974)

Determinism so stated is a supervenience thesis: as Schaffer (2007: 115) puts it, ‘A world \(w\) is deterministic iff: for all times \(t\) in \(w\), the total occurrent history of \(w\) supervenes on the occurrent state of \(w\) at \(t\) together with the laws of \(w\).’

With this in mind, we now evaluate the premises of this argument. There seem to be significant doubts about both directions of both premises. The upshot is that indeterminism offers little comfort for the defender of RCT.

It is very nearly a piece of philosophical orthodoxy that non-trivial objective chances require indeterminism. The view is seldom defended; even those who trouble to state the view explicitly (Lewis, 1980: 120) don’t go to the further effort of justifying it, perhaps because it seems obvious. After all, under determinism, someone who knew the past and the laws would be in a position to know with certainty every future outcome. So our use of probability under determinism must be purely subjective, a side-effect of our (perhaps unavoidable) ignorance of the past or the laws. If this orthodoxy is correct, at least the left-to-right direction of P1 would be true.

However, there has recently been a considerable amount of work by philosophers defending the thesis that chance and determinism are consistent. Many of the topics dealt with above feature in their arguments. Loewer (2001) draws on the best systems analysis of chance to argue that, in worlds like ours (with entropy-increasing processes, and apparently fair coin tosses, etc.), the best description involves some probabilistic component which deserves to be called chance. Clark (1987) draws on how we use the phase space measure \(\mu\) to govern our expectations about the behaviour of Bernoulli and mixing systems in classical statistical mechanics, and argues (in effect) that this is an objective probability function despite the deterministic underlying physics. Various other proposals for deterministic chance have been developed (Eagle 2011; Glynn, 2010; Hoefer, 2007; Ismael, 2009; Sober, 2010). The general technique is to argue that there are probability distributions over outcomes that can play the chance role, even in impeccably deterministic theories. Many of these philosophers are sympathetic to reductionism about chance, which permits the compatibility of chance and determinism. For the fact that the entire history of a world supervenes on any moment of its history, as determinism states, apparently entails nothing one way or another about whether the best description of that entire history involves chances (this is related to the ‘no-connection’ argument of Schaffer 2007: 115). If there is such a thing as deterministic chance, however, P1 is false.

Anti-reductionists about chance have generally found these arguments less persuasive (Popper 1992; Black 1998; Weiner and Belnap 2006). In particular, it is in many ways hard to reconcile the BCP (and RP) with deterministic chance. Doing so will require that there is a physically possible world (sharing the same laws) which matches ours in occurrent facts up until \(t\) but diverges thereafter; but if determinism is true, such a divergence is not possible. If that world matches ours at any time, it matches it at all times. So, it seems, ‘only an incompatibilist function can fit the RP’ (Schaffer 2007: 130). In any case, the debate over whether deterministic ‘chance’, and reductionism about chance more generally is ongoing (see further the discussion in supplement A ); the status of the left-to-right direction of P1 is at best unsettled.

The same cannot be said for the right-to-left direction. For the discussion in § 5.3 showed that there are indeterministic theories without chances, those where indeterminism is secured by the existence of alternative future possibilities, but where those possibilities collectively do not permit or require a probability distribution over them. A more controversial class of cases of indeterminism without chances comes from those who reject universalism about chance: the thesis ‘that chance of truth applies to any proposition whatever’ (Lewis, 1980: 91). If universalism is false, there may be indeterministic situations where the alternative outcomes are those to which chance does not apply. Von Mises rejected universalism, because he thought that chance applied properly only to ‘mass phenomena’; in an indeterministic world where the indeterministic process occurs only once, for example, the theory of probability does not apply. Hoefer (2007) also holds a view something like this, rejecting chances for those processes which don’t have stable enough outcome patterns. [ 25 ]

We could give an argument for P2 if it could be shown that, from the above definition of determinism, we could conclude (i) that only random sequences could occur under indeterminism, and (ii) that random sequences could only occur under indeterminism. However, both parts of this claim are problematic.

Theorem 8 (Humphreys 1978) . There is a theory which is deterministic in the sense of Montague which has as a model a system which produces sequences which are random in the sense of von Mises/Church.

The proof of this theorem relies on the fact that a theory can be non-trivially deterministic without being computable. There is an arithmetically definable function which governs the evolution of the system over time (in Humphrey’s construction, the instantaneous state of the system is an arithmetically definable function of the time, which ensures that any two models agreeing at one time will agree at all times, guaranteeing determinism). But the function is not effectively computable, so no algorithm can produce the sequence of states that this system goes through. [ 26 ] The physical significance of such uncomputable functions is not clear (though see Pour-El and Richards 1983), but the possibility of a deterministic physics which features such equations of motion is enough to undermine a close connection between randomness and indeterminism. This shows that claim (ii) above is false.

Moreover, since we’ve already seen (§ 4.1 ) that it is possible for a chancy and indeterministic process to produce a non-random sequence of outcomes, and such a sequence would not be random, we also have a counterexample to claim (i). Claim (i) could be saved if we made a more robust claim about what could happen under indeterminism. There is a sense in which, while it is possible that a fair coin could land heads an infinite number of times, it would not . That is, the counterfactual ‘If I tossed the coin an infinite number of times, it wouldn’t land all heads’ is apparently true. There is some question whether the counterfactual really is true; Lewis (1979a) and Williams (2008) argue that it is, while Hawthorne (2005) argues that it is not. But if it is, the way lies open to defend a modified counterfactual version of P2:

P\(2'\) : A possible sequence of outcomes is random iff it is not the case that, were an indeterministic process to be repeated infinitely, it would not produce that sequence of outcomes.

But this is highly controversial; and the problem for claim (ii) would still stand.

If we are to accept this argument, then, we shall have to take P2 as an independent truth about randomness. Analyses of randomness as indeterminism, which take P2 to be analytic, have been given: to their detriment, if the foregoing observations are correct. Hellman (1978: 83) suggests that randomness is ‘roughly interchangeable with “indeterministic”’, while Ekeland (1988: 49) says ‘the main feature of randomness is some degree of independence from the initial conditions. … Better still, if one performs the same experiment twice with the same initial conditions, one may get two different outcomes’.

However, it has been argued that this view of randomness as indeterminism makes it difficult to understand many of the uses of randomness in science (Eagle 2005: §3.2). This view entails that random sampling, and random outcomes in chaotic dynamics, and random mating in population genetics, etc., are not in fact random, despite the plausibility of their being so. It does not apparently require fundamental indeterminism to have a randomised trial, and our confidence in the deliverances of such trials does not depend on our confidence that the trial design involved radioactive decay or some other fundamentally indeterministic process. Indeed, if Bohmians or Everettians are right (an open epistemic possibility), and quantum mechanics is deterministic, this view entails that nothing is actually random, not even the most intuitively compelling cases. This kind of view attributes to scientists a kind of error theory about many of their uses of the term ‘random’, but as yet the philosophical evidence adduced to convict scientists of this pervasive error is not compelling.

One reason for the continuing attractiveness of the thesis that randomness is indeterminism may be the fact that, until quite recently, there has been a tendency for philosophers and other to confuse unpredictability and indeterminism. Laplace’s original conception of determinism was an epistemic one:

[A]n intelligence which could comprehend all the forces by which nature is animated and the respective situation of all the [things which] compose it—an intelligence sufficiently vast to submit these data to analysis—it would embrace in the same formula the movements of the greatest bodies in the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as well as the past, would be present to its eyes. (Laplace 1826: p. 4)

This kind of account still resonates with us, despite the fact that with the Montague-Earman definition we now have a non-epistemic characterisation of determinism. Since random sequences will, almost always, be unpredictable, it is clear why we might then taken them to be indeterministic. But once we keep clear the distinction between predictability and determinism, we should be able to avoid this confusion (Bishop, 2003; Schurz, 1995; Werndl, 2009).

From what we have seen, the commonplace thesis cannot be sustained. It would in many ways have been nice if chance of a process and randomness of its product had gone hand in hand—the epistemology of chance would be much aided if it invariably showed itself in random outputs, and we could have had a tight constraint on what outcomes to expect of a repeated chance process, to say nothing of the further interesting consequences the thesis may have for random sampling or probabilistic explanation, mentioned in the introduction. But the counterexamples to the thesis in §§ 4 – 5 show that it is false, even in its most plausible form. Various attempts to salvage the thesis, by appeal to non-standard accounts of chance or randomness, fail to give us a version of the thesis of much interest or bearing on the issues we had hoped it would illuminate. A final attempt to argue directly for the thesis from the connections between chance, randomness, and determinism also failed, though it does shed light on all three notions. It is safest, therefore, to conclude that chance and randomness, while they overlap in many cases, are separate concepts.

This is not to say that there is no link between KML-randomness and physical chance. The observation of a random sequence of outcomes is a defeasible incentive to inquire into the physical basis of the outcome sequence, and it provides at least a prima facie reason to think that a process is chancy (though recall § 4.5 ). Moreover, if we knew that a process is chancy, we should expect (eventually, with high and increasing probability) a random sequence of outcomes. Conversely, a sequence of outcomes that appears predictable, compressible, and rule-governed will be strong disconfirming evidence for any hypothesis to the effect that the sequence was produced by chance alone. Hellman concludes

The link, then, between mathematical and physical randomness is epistemic and only that. Observations of mathematically non-random sequences can be used to decide when further explanation in terms of as yet undiscovered causal factors is wanting. But, in no sense is any notion of mathematical randomness serving as an explication for ‘ultimate physical randomness’, whatever that might be. (Hellman, 1978: 86)

Taking ‘mathematical randomness’ to be product randomness, and ‘physical randomness’ to mean process randomness (chanciness), this conclusion seems unavoidable.

A parallel with the relationship between frequencies and chances is tempting and unsurprising. Relative frequencies are good but not infallible indicators of the chances, and the existence of outcome frequencies strictly between 0 and 1 is evidence that chance processes are involved in the production of those outcomes. But frequentism is implausible as a reductive account of chance. And it is no more plausible to think that chance is present iff random sequences of outcomes are. [ 27 ] An evidential and epistemic connection between chance and randomness falls well short of the conceptual connection proposed by the Commonplace Thesis with which we started.

• Agafonov, V. N., 1968, ‘Normal sequences and finite automata’, Soviet Mathematics Doklady , 9: 324–25.
• Albert, David Z., 1992, Quantum Mechanics and Experience , Cambridge, MA: Harvard University Press.
• –––, 2000, Time and Chance , Cambridge, MA: Harvard University Press.
• Arntzenius, Frank and Ned Hall, 2003, ‘On What We Know About Chance’, British Journal for the Philosophy of Science , 54: 171–9.
• Bar-Hillel, Maya and Willem A. Wagenaar, 1991, ‘The Perception of Randomness’, Advances in Applied Mathematics , 12: 428–54.
• Barrett, Jeffrey A., 1999, The Quantum Mechanics of Minds and Worlds , Oxford: Oxford University Press.
• Beall, JC and Greg Restall, 2006, Logical Pluralism , Oxford: Oxford University Press.
• Bell, John S., 1964, ‘On the Einstein-Podolsky-Rosen paradox’, Physics , 1: 195–200.
• Berkovitz, Joseph, Roman Frigg, and Fred Kronz, 2006, ‘The Ergodic Hierarchy, Randomness and Chaos’, Studies in History and Philosophy of Modern Physics 37: 661–91.
• Bigelow, John, John Collins and Robert Pargetter, 1993, ‘The Big Bad Bug: What are the Humean’s Chances?’, British Journal for the Philosophy of Science , 44: 443–62.
• Bishop, Robert C., 2003, ‘On Separating Predictability and Determinism’, Erkenntnis , 58: 169–88.
• Black, Robert, 1998, ‘Chance, Credence, and the Principal Principle’, British Journal for the Philosophy of Science , 49(3): 371–85.
• Boolos, George S., John P. Burgess, and Richard C. Jeffrey, 2003, Computability and Logic , Cambridge: Cambridge University Press, 4th edition.
• Borel, Émile, 1909, ‘Les Probabilités Dénombrables et Leurs Applications Arithmétiques’, Rendiconti del Circolo Matematico di Palermo , 27: 247–71.
• Butler, Joseph, 1736, The Analogy of Religion, Natural and Revealed, to the Constitution and Course of Nature , London: Knapton.
• Callender, Craig, 2009, ‘Thermodynamic Asymmetry in Time’, in The Stanford Encyclopedia of Philosophy (Spring 2009 edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/spr2009/entries/time-thermo/ \(\gt\).
• Carnap, Rudolf, 1945, ‘The Two Concepts of Probability’, Philosophy and Phenomenological Research , 5: 513–32.
• Chaitin, Gregory, 1966, ‘On the Length of Programs for Computing Finite Binary Sequences’, Journal of the Association for Computing Machinery , 13: 547–69.
• –––, 1975, ‘Randomness and Mathematical Proof’, Scientific American , 232: 47–52.
• –––, 1987, ‘Incompleteness Theorems for Random Reals’, Advances in Applied Mathematics , 8: 119–46.
• Champernowne, D. G., 1933, ‘The Construction of Decimals Normal in the Scale of Ten’, Journal of the London Mathematical Society , 8: 254–60.
• Church, Alonzo, 1940, ‘On the Concept of a Random Sequence’, Bulletin of the American Mathematical Society , 46: 130–135.
• Clark, Peter, 1987, ‘Determinism and Probability in Physics’, Proceedings of the Aristotelian Society, Supplementary Volume , 61: 185–210.
• Dasgupta, Abhijit, 2011, ‘Mathematical Foundations of Randomness’, in Prasanta Bandyopadhyay and Malcolm Forster (eds.), Philosophy of Statistics (Handbook of the Philosophy of Science: Volume 7), Amsterdam: Elsevier, pp. 641–710.
• Davies, Martin and Lloyd Humberstone, 1980, ‘Two Notions of Necessity’, Philosophical Studies , 38: 1–30.
• de Finetti, Bruno, 1964, ‘Foresight: Its Logical Laws, Its Subjective Sources’, In Studies in Subjective Probability , Henry E. Kyburg, Jr. and Howard E. Smokler (eds.), New York: Wiley, pp. 93–158.
• Delahaye, Jean-Paul, 1993, ‘Randomness, Unpredictability and Absence of Order’, In Philosophy of Probability , Jacques-Paul Dubucs (ed.), Dordrecht: Kluwer, pp. 145–67.
• Doob, J. L., 1936, ‘Note on Probability’, Annals of Mathematics , 37: 363–7.
• Downey, Rod and E. Griffiths, 2004, ‘Schnorr randomness’, Journal of Symbolic Logic , 69: 533–54.
• Downey, Rod and Denis R. Hirschfeldt, 2010, Algorithmic Randomness and Complexity , Berlin: Springer.
• –––, –––, André Nies and Sebastiaan A. Terwijn, 2006, ‘Calibrating Randomness’, Bulletin of Symbolic Logic , 12: 411–91.
• Eagle, Antony, 2004, ‘Twenty-One Arguments Against Propensity Analyses of Probability’, Erkenntnis , 60: 371–416.
• –––, 2005, ‘Randomness is Unpredictability’, British Journal for the Philosophy of Science , 56: 749–90.
• –––, 2011, ‘Deterministic Chance’, Noûs , 45: 269–99.
• –––, 2016, ‘Probability and Randomness’, In The Oxford Handbook of Probability and Philosophy , Alan Hájek and Christopher Hitchcock (eds.), Oxford: Oxford University Press, pp. 440–59.
• Earman, John, 1986, A Primer on Determinism , Dordrecht: D. Reidel.
• Eells, Ellery, 1983, ‘Objective Probability Theory Theory’, Synthese , 57: 387–442.
• Ekeland, Ivar, 1988, Mathematics and the Unexpected , Chicago: University of Chicago Press.
• Elga, Adam, 2004, ‘Infinitesimal Chances and the Laws of Nature’, Australasian Journal of Philosophy , 82: 67–76.
• Feller, William, 1945, ‘The Fundamental Limit Theorems in Probability’, Bulletin of the American Mathematical Society , 51: 800–32.
• –––, 1950, An Introduction to Probability Theory and Its Applications . New York: Wiley.
• Frigg, Roman, 2004, ‘In What Sense is the Kolmogorov-Sinai Entropy a Measure for Chaotic Behaviour?—Bridging the Gap Between Dynamical Systems Theory and Communication Theory’, British Journal for the Philosophy of Science , 55: 411–34.
• Futuyma, Douglas J., 2005, Evolution . Cumberland, MA: Sinauer.
• Gaifman, Haim, 1988, ‘A Theory of Higher Order Probabilities’, in Causation, Chance and Credence , volume 1, Brian Skyrms and William Harper (eds.), Dordrecht: Kluwer, pp. 191–219.
• –––, and Marc Snir, 1982, ‘Probabilities Over Rich Languages, Testing and Randomness’, Journal of Symbolic Logic , 47: 495–548.
• Gates, P. and H. Tong, 1976, ‘On Markov Chain Modeling to Some Weather Data’, Journal of Applied Meteorology , 15: 1145–51.
• Giere, Ronald N., 1973, ‘Objective Single-Case Probabilities and the Foundations of Statistics’, 4, Amsterdam: North-Holland, 467–83.
• Gillies, Donald, 2000, ‘Varieties of Propensity’, British Journal for the Philosophy of Science , 51: 807–35.
• Gilovich, Thomas, Robert Vallone, and Amos Tversky, 1985, ‘The Hot Hand in Basketball: On the misperception of random sequences’, Cognitive Psychology , 17: 295–314.
• Glynn, Luke, 2010, ‘Deterministic Chance’, British Journal for the Philosophy of Science , 61: 51–80.
• Gowers, Timothy, (ed.), 2008, The Princeton Companion to Mathematics , Princeton: Princeton University Press.
• Hacking, Ian, 1965, The Logic of Statistical Inference , Cambridge: Cambridge University Press.
• Hahn, Ulrike and Paul A Warren, 2009, ‘Perceptions of Randomness: Why Three Heads Are Better Than Four’, Psychological Review , 116: 454–61.
• Hájek, Alan, 1997, ‘“Mises Redux”-Redux: Fifteen Arguments Against Finite Frequentism’, Erkenntnis , 45: 209–227.
• –––, 2007, ‘The Reference Class Problem is Your Problem Too’, Synthese , 156: 563–85.
• –––, 2009, ‘Fifteen Arguments Against Hypothetical Frequentism’, Erkenntnis , 70: 211–235.
• –––, 2012, ‘Interpretations of Probability’, in The Stanford Encyclopedia of Philosophy (Winter 2012 edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/win2012/entries/probability-interpret/ \(\gt\).
• Hall, Ned, 1994, ‘Correcting the Guide to Objective Chance’, Mind , 103: 505–17.
• –––, 2004, ‘Two Mistakes About Credence and Chance’, in Lewisian Themes , Frank Jackson and Graham Priest (eds.), Oxford: Oxford University Press, pp. 94–112.
• Hawthorne, John, 2005, ‘Chance and Counterfactuals’, Philosophy and Phenomenological Research , 70: 396–405.
• ––– and Maria Lasonen-Aarnio, 2009, ‘Knowledge and Objective Chance’, in Williamson on Knowledge , Patrick Greenough and Duncan Pritchard (eds.), Oxford: Oxford University Press, pp. 92–108.
• Hellman, Geoffrey, 1978, ‘Randomness and Reality’, in PSA 1978 , volume 2, Peter D. Asquith and Ian Hacking (eds.), Chicago: University of Chicago Press, pp. 79–97.
• Hoefer, Carl, 2007, ‘The Third Way on Objective Probability: A Sceptic’s Guide to Objective Chance’, Mind , 116: 549–96.
• –––, 2010, ‘Causal Determinism’, The Stanford Encyclopedia of Philosophy (Spring 2010 Edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/spr2010/entries/determinism-causal/ \(\gt\).
• Howson, Colin and Peter Urbach, 1993, Scientific Reasoning: the Bayesian Approach , Chicago: Open Court, 2nd edition.
• Humphreys, Paul W., 1978, ‘Is “Physical Randomness” Just Indeterminism in Disguise?’, in PSA 1978 , volume 2, Peter D. Asquith and Ian Hacking (eds.), Chicago: University of Chicago Press, pp. 98–113.
• –––, 1985, ‘Why Propensities Cannot be Probabilities’, Philosophical Review , 94: 557–70.
• Ismael, Jenann, 1996, ‘What Chances Could Not Be’, British Journal for the Philosophy of Science , 47: 79–91.
• –––, 2008, ‘Raid! Dissolving the Big, Bad Bug’, Noûs , 42: 292–307.
• –––, 2009, ‘Probability in Deterministic Physics’, Journal of Philosophy , 106: 89–109.
• Jeffrey, Richard C., 1977, ‘Mises Redux’, in Basic Problems in Methodology and Linguistics , Robert E. Butts and Jaakko Hintikka (eds.), Dordrecht: D. Reidel, pp. 213–222.
• Joyce, James M., 1998, ‘A Nonpragmatic Vindication of Probabilism’, Philosophy of Science , 65: 575–603.
• –––, 2007, ‘Epistemic Deference: The Case of Chance’, Proceedings of the Aristotelian Society , 107: 187–206.
• Kahneman, Daniel and Amos Tversky, 1972, ‘Subjective Probability: A judgment of representativeness’, Cognitive Psychology , 3: 430–54.
• Kautz, Steven M., 1991, Degrees of random sets , Ph.D. thesis, Cornell University, [ Available online ].
• Kolmogorov, A. N., 1929, Über das Gesetz des iterierten Logarithmus’, Math. Ann. , 101: 126–35.
• –––, 1933, Grundbegriffe der Wahrscheinlichkeitrechnung , Ergebnisse Der Mathematik; translated as Foundations of Probability , Chelsea Publishing Company, 1950.
• –––, 1963, ‘On Tables of Random Numbers’, Sankhya , 25: 369–376.
• –––, 1965, ‘Three Approaches to the Definition of the Concept “Quantity of Information”’. Problemy Peredachi Informatsii , 1: 3–11.
• –––, and V. A. Uspensky, 1988, ‘Algorithms and Randomness’. SIAM Theory of Probability and Applications , 32: 389–412.
• Kratzer, Angelika, 1977, ‘What ‘must’ and ‘can’ Must and Can Mean’. Linguistics and Philosophy , 1: 337–55.
• Kyburg, Jr., Henry E., 1974, The Logical Foundations of Statistical Inference , Dordrecht: D. Reidel.
• –––, 1978, ‘Subjective probability: Criticisms, reflections, and problems’, Journal of Philosophical Logic , 7: 157–80.
• Laplace, Pierre-Simon, 1826, Philosophical Essay on Probabilities , New York: Dover, 1951
• Levi, Isaac, 1980, The Enterprise of Knowledge , Cambridge, MA: MIT Press.
• Levin, L. A. and A. K. Zvonkin, 1970, ‘The Complexity of Finite Objects and the Basing of the Concepts of Information and Randomness on the Theory of Algorithms’, Uspekhi Matematicheskikh Nauk , 25 (6 (156)): 85–127.
• Lewis, David, 1973, Counterfactuals , Oxford: Blackwell’s.
• –––, 1979a, ‘Counterfactual Dependence and Time’s Arrow’, in his Philosophical Papers , volume 2, Oxford: Oxford University Press, 1986, pp. 32–66.
• –––, 1979b, ‘Scorekeeping in a Language Game’, Journal of Philosophical Logic , 8: 339–59.
• –––, 1980, ‘A Subjectivist’s Guide to Objective Chance’, in his Philosophical Papers , volume 2, Oxford: Oxford University Press, 1986, pp. 83–132.
• –––, 1994, ‘Humean Supervenience Debugged’, Mind , 103: 473–90.
• Li, Ming and Paul M. B. Vitányi, 2008, An Introduction to Kolmogorov Complexity and Its Applications , Berlin and New York: Springer Verlag, 3rd edition.
• Loewer, Barry, 2001, ‘Determinism and Chance’, Studies in History and Philosophy of Modern Physics , 32: 609–20.
• –––, 2004, ‘David Lewis’ Humean Theory of Objective Chance’, Philosophy of Science , 71: 1115–25.
• –––, 2007, ‘Counterfactuals and the Second Law’, in Causation, Physics, and the Constitution of Reality: Russell’s Republic Revisited , Huw Price and Richard Corry (eds.), Oxford: Oxford University Press, pp. 293–326.
• Luzzatto, Stefano, Ian Melbourne, and Frederic Paccaut, 2005, ‘The Lorenz Attractor is Mixing’, Communications in Mathematical Physics , 260: 393–401, [ Preprint available online ].
• Martin-Löf, Per, 1966, ‘The Definition of a Random Sequence’, Information and Control , 9: 602–619.
• –––, 1969a, ‘Algorithms and Randomness’, Review of the International Statistical Institute , 37: 265–72.
• –––, 1969b, ‘The Literature on von Mises’ Kollektivs Revisited’, Theoria , 35: 12–37.
• Mather, J. and R. McGehee, 1975, ‘Solutions of the collinear four-body problem which become unbounded in finite time’, in Dynamical Systems, Theory and Applications , (Series: Lecture Notes in Physics, Volume 38), J. Moser (ed.), Berlin: Springer Verlag, pp. 573–97.
• Matsumoto, Makoto and Takuji Nishimura, 1998, ‘Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator’, ACM Transactions on Modeling and Computer Simulation , 8(1): 3–30.
• Mellor, D. H., 2000, ‘Possibility, Chance and Necessity’, Australasian Journal of Philosophy , 78: 16–27.
• Milne, P., 1985, ‘Can there be a Realist Single Case Interpretation of Probability?’ Erkenntnis , 25: 129–32.
• Montague, Richard, 1974, ‘Deterministic Theories’, in Formal Philosophy , Richmond H. Thomason (ed.), New Haven: Yale University Press.
• Nies, André, 2009, Computability and Randomness , Oxford: Oxford University Press.
• Norton, John D., 2003, ‘Causation as Folk Science’, Philosophers’ Imprint , 3(4), URL = [ Available online ].
• –––, 2008, ‘The Dome: An Unexpectedly Simple Failure of Determinism’, Philosophy of Science , 75: 786–98.
• Pathak, Noopur, 2009, ‘A computational aspect of the Lebesgue differentiation theorem’, Journal of Logic and Analysis , 1 (9): 1–15 [ Available online ].
• Popper, Karl, 1959, ‘A Propensity Interpretation of Probability’, British Journal for the Philosophy of Science , 10: 25–42.
• –––, 1992, Quantum Theory and the Schism in Physics , New York: Routledge.
• Porter, Christopher P., 2016, ‘On Analogues of the Church-Turing Thesis in Algorithmic Randomness’, The Review of Symbolic Logic , 9 (3): 456–79.
• Pour-El, B. M. and I. Richards, 1983, ‘Non-computability in Analysis and Physics’, Advances in Mathematics , 48: 44–74.
• RAND Corporation, 1955, A Million Random Digits with 100,000 Normal Deviates , New York: Free Press.
• Reichenbach, Hans, 1949, The Theory of Probability , Berkeley: University of California Press.
• Saari, Donald G. and Zhihong (Jeff) Xia, 1995, ‘Off to Infinity in Finite Time’, Notices of the American Mathematical Society , 42: 538–46.
• Salmon, Wesley C., 1977, ‘Objectively Homogeneous Reference Classes’, Synthese , 36: 399–414.
• Schaffer, Jonathan, 2003, ‘Principled Chances’, British Journal for the Philosophy of Science , 54: 27–41.
• –––, 2007, ‘Deterministic Chance?’ British Journal for the Philosophy of Science , 58: 113–40.
• Schnorr, C. P., 1971, ‘A unified approach to the definition of random sequences’, Theory of Computing Systems , 5: 246–58.
• –––, 1977, ‘A survey of the theory of random sequences’, in R. E. Butts and J. Hintikka (eds.), Basic Problems in Methodology and Linguistics , Dordrecht: D. Reidel, pp. 193–210.
• Schurz, Gerhard, 1995, ‘Kinds of Unpredictability in Deterministic Systems’, in Law and Prediction in the Light of Chaos Research , P. Weingartner and G. Schurz (eds.), Berlin: Springer, pp. 123–41.
• Schwarz, Wolfgang, 2016, ‘Best System Approaches to Chance’, In The Oxford Handbook of Probability and Philosophy , Alan Hájek and Christopher Hitchcock (eds.), Oxford: Oxford University Press, pp. 423–39.
• Shimony, Abner, 2009, ‘Bells’ Theorem’, in The Stanford Encyclopedia of Philosophy (Summer 2009 edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/sum2009/entries/bell-theorem/ \(\gt\)
• Sinai, Yakov G., 1992, Probability Theory , Berlin and Heidelberg: Springer-Verlag. Translated by D. Haughton.
• Sklar, Lawrence, 1993, Physics and Chance , Cambridge: Cambridge University Press.
• Skyrms, Brian, 1980, Causal Necessity , New Haven: Yale University Press.
• Smith, Peter, 1998, Explaining Chaos . Cambridge: Cambridge University Press.
• Sober, Elliott, 2010, ‘Evolutionary Theory and the Reality of Macro Probabilities’. in Probability in Science , Ellery Eells and James H. Fetzer (eds.), Dordrecht: Springer, 133–61, [ Preprint available online ]
• Stalnaker, Robert, 1978, ‘Assertion’. in P. Cole (ed), Syntax and Semantics 9 , New York: New York Academic Press, 315–32.
• Strevens, Michael, 1999, ‘Objective Probability as a Guide to the World’, Philosophical Studies , 95: 243–75.
• Suppes, Patrick, 1984, Probabilistic Metaphysics , Oxford: Blackwell.
• Talbott, William, 2008, ‘Bayesian Epistemology’, in The Stanford Encyclopedia of Philosophy (Fall 2008 edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/fall2008/entries/epistemology-bayesian/ \(\gt\).
• Thau, Michael, 1994, ‘Undermining and Admissibility’, Mind , 103: 491–503.
• van Lambalgen, Michiel, 1987a, Random Sequences , Ph.D. thesis, University of Amsterdam, [Available online] .
• –––, 1987b, ‘Von Mises’ Definition of Random Sequences Revisited’, Journal of Symbolic Logic , 52: 725–55.
• –––, 1995, ‘Randomness and Infinity’, Tech. Rep. ML-1995-01, ILLC, University of Amsterdam, URL = [ Available online (in compressed Postscript)].
• Venn, John, 1876, The Logic of Chance , London: Macmillan and Co., 2nd edition.
• Ville, J., 1939, Étude Critique de la Notion Collectif , Paris: Gauthier-Villars.
• von Mises, Richard, 1941, ‘On the Foundations of Probability and Statistics’, Annals of Mathematical Statistics , 12: 191–205.
• ––– 1957, Probability, Statistics and Truth . New York: Dover.
• Wald, Abraham, 1938, ‘Der Widerspruchsfreiheit des Kollektivbegriffes’, Actualités Scientifique et Industrielles , 735: 79–99.
• Wallace, David, 2007, ‘Quantum Probability from Subjective Likelihood: Improving on Deutsch’s Proof of the Probability Rule’. Studies in History and Philosophy of Modern Physics , 38: 311–32.
• Wang, Yongge, 1996, Randomness and Complexity , Ph.D. Thesis, University of Heidelberg, Available online ].
• Weatherson, Brian, 2010 ‘David Lewis’, The Stanford Encyclopedia of Philosophy (Summer 2010 Edition), Edward N. Zalta (ed.), URL \(= \lt\) https://plato.stanford.edu/archives/sum2010/entries/david-lewis/ \(\gt\).
• Weiner, Matthew and Nuel Belnap, 2006, ‘How Causal Probabilities Might Fit Into Our Objectively Indeterministic World’, Synthese 149(1): 1–36.
• Werndl, Charlotte, 2009, ‘What Are the New Implications of Chaos for Unpredictability?’ British Journal for the Philosophy of Science , 60: 195–220.
• Williams, J. R. G., 2008, ‘Chances, Counterfactuals, and Similarity’, Philosophy and Phenomenological Research , 77: 385–420.
• Williamson, Timothy (2006), ‘Indicative Versus Subjunctive Conditionals, Congruential Versus Non-Hyperintensional Contexts’, Philosophical Issues , 16: 310–33.
• –––, 2007, ‘How Probable is an Infinite Sequence of Heads?’ Analysis , 67: 173–80.

How to cite this entry . Preview the PDF version of this entry at the Friends of the SEP Society . Look up topics and thinkers related to this entry at the Internet Philosophy Ontology Project (InPhO). Enhanced bibliography for this entry at PhilPapers , with links to its database.

• Brattka, Vasco, Joseph S. Miller and André Nies, 2011, ‘Randomness and Differentiability’ arXiv:1104.4465 [Available online]
• Hájek, Alan, in progress, Most Counterfactuals are False , manuscript. [Available online]
• Kautz, S., 1991, Degrees of Random Sets , Ph.D. Thesis, Cornell University.
• Lieb, E. H., Daniel Osherson, and Scott Weinstein, 2006, ‘Elementary proof of a theorem of Jean Ville’ arXiv:cs/0607054v1 [Available online] .
• Nies, André, 2011, ‘Randomness and computable analysis: Results and open questions’, talk at Computability, Complexity and Randomness, Cape Town, [Slides Available Online] .
• Norton’s Dome
• Reaber, Grant, 2010, ‘ The Deference-Based Conception of Rational Belief Updating ’.
• Reimann, Jan, and Theodore A. Slaman, 2008, ‘ Measures and Their Random Reals .’
• Rute, Jason, 2011, ‘Randomness and the Lebesgue Differentiation Theorem’, talk at Southern Wisconsin Logic Colloquium, [Slides available online] .

Bell’s Theorem | chaos | computability and complexity | determinism: causal | epistemology: Bayesian | Lewis, David | probability, interpretations of | statistical physics: philosophy of statistical mechanics | time: thermodynamic asymmetry in

Acknowledgments

Thanks to audiences at the Sigma Group at the LSE, Leeds HPS, and the first year seminar in Oxford, for comments on presentations of parts of this material, and to Alan Hájek, Chris Porter, and Fred Kroon for extensive and very helpful comments on a draft entry. (In particular, the argument of supplement A.2 , note 4 is due to Hájek.) In revising the entry, I’ve been grateful to Chris Porter for further helpful comments and pointers to some of the recent literature. The ‘pluralist’ approach mentioned in supplement B.1.2 is due to him, as in the broad outlines of the argument of B.1.3 .

Copyright © 2018 by Antony Eagle < antony . eagle @ adelaide . edu . au >

• Accessibility

Support SEP

Mirror sites.

View this site from another server:

• Info about mirror sites

The Stanford Encyclopedia of Philosophy is copyright © 2023 by The Metaphysics Research Lab , Department of Philosophy, Stanford University

Library of Congress Catalog Data: ISSN 1095-5054

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

How to Write a Great Hypothesis

Hypothesis Definition, Format, Examples, and Tips

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Amy Morin, LCSW, is a psychotherapist and international bestselling author. Her books, including "13 Things Mentally Strong People Don't Do," have been translated into more than 40 languages. Her TEDx talk,  "The Secret of Becoming Mentally Strong," is one of the most viewed talks of all time.

Verywell / Alex Dos Diaz

• The Scientific Method

Hypothesis Format

Falsifiability of a hypothesis.

• Operationalization

Hypothesis Types

Hypotheses examples.

• Collecting Data

A hypothesis is a tentative statement about the relationship between two or more variables. It is a specific, testable prediction about what you expect to happen in a study. It is a preliminary answer to your question that helps guide the research process.

Consider a study designed to examine the relationship between sleep deprivation and test performance. The hypothesis might be: "This study is designed to assess the hypothesis that sleep-deprived people will perform worse on a test than individuals who are not sleep-deprived."

At a Glance

A hypothesis is crucial to scientific research because it offers a clear direction for what the researchers are looking to find. This allows them to design experiments to test their predictions and add to our scientific knowledge about the world. This article explores how a hypothesis is used in psychology research, how to write a good hypothesis, and the different types of hypotheses you might use.

The Hypothesis in the Scientific Method

In the scientific method , whether it involves research in psychology, biology, or some other area, a hypothesis represents what the researchers think will happen in an experiment. The scientific method involves the following steps:

• Forming a question
• Performing background research
• Creating a hypothesis
• Designing an experiment
• Collecting data
• Analyzing the results
• Drawing conclusions
• Communicating the results

The hypothesis is a prediction, but it involves more than a guess. Most of the time, the hypothesis begins with a question which is then explored through background research. At this point, researchers then begin to develop a testable hypothesis.

Unless you are creating an exploratory study, your hypothesis should always explain what you  expect  to happen.

In a study exploring the effects of a particular drug, the hypothesis might be that researchers expect the drug to have some type of effect on the symptoms of a specific illness. In psychology, the hypothesis might focus on how a certain aspect of the environment might influence a particular behavior.

Remember, a hypothesis does not have to be correct. While the hypothesis predicts what the researchers expect to see, the goal of the research is to determine whether this guess is right or wrong. When conducting an experiment, researchers might explore numerous factors to determine which ones might contribute to the ultimate outcome.

In many cases, researchers may find that the results of an experiment  do not  support the original hypothesis. When writing up these results, the researchers might suggest other options that should be explored in future studies.

In many cases, researchers might draw a hypothesis from a specific theory or build on previous research. For example, prior research has shown that stress can impact the immune system. So a researcher might hypothesize: "People with high-stress levels will be more likely to contract a common cold after being exposed to the virus than people who have low-stress levels."

In other instances, researchers might look at commonly held beliefs or folk wisdom. "Birds of a feather flock together" is one example of folk adage that a psychologist might try to investigate. The researcher might pose a specific hypothesis that "People tend to select romantic partners who are similar to them in interests and educational level."

Elements of a Good Hypothesis

So how do you write a good hypothesis? When trying to come up with a hypothesis for your research or experiments, ask yourself the following questions:

• Is your hypothesis based on your research on a topic?
• Can your hypothesis be tested?
• Does your hypothesis include independent and dependent variables?

Before you come up with a specific hypothesis, spend some time doing background research. Once you have completed a literature review, start thinking about potential questions you still have. Pay attention to the discussion section in the  journal articles you read . Many authors will suggest questions that still need to be explored.

How to Formulate a Good Hypothesis

To form a hypothesis, you should take these steps:

• Collect as many observations about a topic or problem as you can.
• Evaluate these observations and look for possible causes of the problem.
• Create a list of possible explanations that you might want to explore.
• After you have developed some possible hypotheses, think of ways that you could confirm or disprove each hypothesis through experimentation. This is known as falsifiability.

In the scientific method ,  falsifiability is an important part of any valid hypothesis. In order to test a claim scientifically, it must be possible that the claim could be proven false.

Students sometimes confuse the idea of falsifiability with the idea that it means that something is false, which is not the case. What falsifiability means is that  if  something was false, then it is possible to demonstrate that it is false.

One of the hallmarks of pseudoscience is that it makes claims that cannot be refuted or proven false.

The Importance of Operational Definitions

A variable is a factor or element that can be changed and manipulated in ways that are observable and measurable. However, the researcher must also define how the variable will be manipulated and measured in the study.

Operational definitions are specific definitions for all relevant factors in a study. This process helps make vague or ambiguous concepts detailed and measurable.

For example, a researcher might operationally define the variable " test anxiety " as the results of a self-report measure of anxiety experienced during an exam. A "study habits" variable might be defined by the amount of studying that actually occurs as measured by time.

These precise descriptions are important because many things can be measured in various ways. Clearly defining these variables and how they are measured helps ensure that other researchers can replicate your results.

Replicability

One of the basic principles of any type of scientific research is that the results must be replicable.

Replication means repeating an experiment in the same way to produce the same results. By clearly detailing the specifics of how the variables were measured and manipulated, other researchers can better understand the results and repeat the study if needed.

Some variables are more difficult than others to define. For example, how would you operationally define a variable such as aggression ? For obvious ethical reasons, researchers cannot create a situation in which a person behaves aggressively toward others.

To measure this variable, the researcher must devise a measurement that assesses aggressive behavior without harming others. The researcher might utilize a simulated task to measure aggressiveness in this situation.

Hypothesis Checklist

• Does your hypothesis focus on something that you can actually test?
• Does your hypothesis include both an independent and dependent variable?
• Can you manipulate the variables?
• Can your hypothesis be tested without violating ethical standards?

The hypothesis you use will depend on what you are investigating and hoping to find. Some of the main types of hypotheses that you might use include:

• Simple hypothesis : This type of hypothesis suggests there is a relationship between one independent variable and one dependent variable.
• Complex hypothesis : This type suggests a relationship between three or more variables, such as two independent and dependent variables.
• Null hypothesis : This hypothesis suggests no relationship exists between two or more variables.
• Alternative hypothesis : This hypothesis states the opposite of the null hypothesis.
• Statistical hypothesis : This hypothesis uses statistical analysis to evaluate a representative population sample and then generalizes the findings to the larger group.
• Logical hypothesis : This hypothesis assumes a relationship between variables without collecting data or evidence.

A hypothesis often follows a basic format of "If {this happens} then {this will happen}." One way to structure your hypothesis is to describe what will happen to the  dependent variable  if you change the  independent variable .

The basic format might be: "If {these changes are made to a certain independent variable}, then we will observe {a change in a specific dependent variable}."

A few examples of simple hypotheses:

• "Students who eat breakfast will perform better on a math exam than students who do not eat breakfast."
• "Students who experience test anxiety before an English exam will get lower scores than students who do not experience test anxiety."​
• "Motorists who talk on the phone while driving will be more likely to make errors on a driving course than those who do not talk on the phone."
• "Children who receive a new reading intervention will have higher reading scores than students who do not receive the intervention."

Examples of a complex hypothesis include:

• "People with high-sugar diets and sedentary activity levels are more likely to develop depression."
• "Younger people who are regularly exposed to green, outdoor areas have better subjective well-being than older adults who have limited exposure to green spaces."

Examples of a null hypothesis include:

• "There is no difference in anxiety levels between people who take St. John's wort supplements and those who do not."
• "There is no difference in scores on a memory recall task between children and adults."
• "There is no difference in aggression levels between children who play first-person shooter games and those who do not."

Examples of an alternative hypothesis:

• "People who take St. John's wort supplements will have less anxiety than those who do not."
• "Adults will perform better on a memory task than children."
• "Children who play first-person shooter games will show higher levels of aggression than children who do not." 

Collecting Data on Your Hypothesis

Once a researcher has formed a testable hypothesis, the next step is to select a research design and start collecting data. The research method depends largely on exactly what they are studying. There are two basic types of research methods: descriptive research and experimental research.

Descriptive Research Methods

Descriptive research such as  case studies ,  naturalistic observations , and surveys are often used when  conducting an experiment is difficult or impossible. These methods are best used to describe different aspects of a behavior or psychological phenomenon.

Once a researcher has collected data using descriptive methods, a  correlational study  can examine how the variables are related. This research method might be used to investigate a hypothesis that is difficult to test experimentally.

Experimental Research Methods

Experimental methods  are used to demonstrate causal relationships between variables. In an experiment, the researcher systematically manipulates a variable of interest (known as the independent variable) and measures the effect on another variable (known as the dependent variable).

Unlike correlational studies, which can only be used to determine if there is a relationship between two variables, experimental methods can be used to determine the actual nature of the relationship—whether changes in one variable actually  cause  another to change.

The hypothesis is a critical part of any scientific exploration. It represents what researchers expect to find in a study or experiment. In situations where the hypothesis is unsupported by the research, the research still has value. Such research helps us better understand how different aspects of the natural world relate to one another. It also helps us develop new hypotheses that can then be tested in the future.

Thompson WH, Skau S. On the scope of scientific hypotheses .  R Soc Open Sci . 2023;10(8):230607. doi:10.1098/rsos.230607

Taran S, Adhikari NKJ, Fan E. Falsifiability in medicine: what clinicians can learn from Karl Popper [published correction appears in Intensive Care Med. 2021 Jun 17;:].  Intensive Care Med . 2021;47(9):1054-1056. doi:10.1007/s00134-021-06432-z

Eyler AA. Research Methods for Public Health . 1st ed. Springer Publishing Company; 2020. doi:10.1891/9780826182067.0004

Nosek BA, Errington TM. What is replication ?  PLoS Biol . 2020;18(3):e3000691. doi:10.1371/journal.pbio.3000691

Aggarwal R, Ranganathan P. Study designs: Part 2 - Descriptive studies .  Perspect Clin Res . 2019;10(1):34-36. doi:10.4103/picr.PICR_154_18

Nevid J. Psychology: Concepts and Applications. Wadworth, 2013.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

What Is a Hypothesis? (Science)

If...,Then...

Angela Lumsden/Getty Images

• Scientific Method
• Chemical Laws
• Periodic Table
• Projects & Experiments
• Biochemistry
• Physical Chemistry
• Medical Chemistry
• Chemistry In Everyday Life
• Famous Chemists
• Activities for Kids
• Abbreviations & Acronyms
• Weather & Climate
• Ph.D., Biomedical Sciences, University of Tennessee at Knoxville
• B.A., Physics and Mathematics, Hastings College

A hypothesis (plural hypotheses) is a proposed explanation for an observation. The definition depends on the subject.

In science, a hypothesis is part of the scientific method. It is a prediction or explanation that is tested by an experiment. Observations and experiments may disprove a scientific hypothesis, but can never entirely prove one.

In the study of logic, a hypothesis is an if-then proposition, typically written in the form, "If X , then Y ."

In common usage, a hypothesis is simply a proposed explanation or prediction, which may or may not be tested.

Writing a Hypothesis

Most scientific hypotheses are proposed in the if-then format because it's easy to design an experiment to see whether or not a cause and effect relationship exists between the independent variable and the dependent variable . The hypothesis is written as a prediction of the outcome of the experiment.

Null Hypothesis and Alternative Hypothesis

Statistically, it's easier to show there is no relationship between two variables than to support their connection. So, scientists often propose the null hypothesis . The null hypothesis assumes changing the independent variable will have no effect on the dependent variable.

In contrast, the alternative hypothesis suggests changing the independent variable will have an effect on the dependent variable. Designing an experiment to test this hypothesis can be trickier because there are many ways to state an alternative hypothesis.

For example, consider a possible relationship between getting a good night's sleep and getting good grades. The null hypothesis might be stated: "The number of hours of sleep students get is unrelated to their grades" or "There is no correlation between hours of sleep and grades."

An experiment to test this hypothesis might involve collecting data, recording average hours of sleep for each student and grades. If a student who gets eight hours of sleep generally does better than students who get four hours of sleep or 10 hours of sleep, the hypothesis might be rejected.

But the alternative hypothesis is harder to propose and test. The most general statement would be: "The amount of sleep students get affects their grades." The hypothesis might also be stated as "If you get more sleep, your grades will improve" or "Students who get nine hours of sleep have better grades than those who get more or less sleep."

In an experiment, you can collect the same data, but the statistical analysis is less likely to give you a high confidence limit.

Usually, a scientist starts out with the null hypothesis. From there, it may be possible to propose and test an alternative hypothesis, to narrow down the relationship between the variables.

Example of a Hypothesis

Examples of a hypothesis include:

• If you drop a rock and a feather, (then) they will fall at the same rate.
• Plants need sunlight in order to live. (if sunlight, then life)
• Eating sugar gives you energy. (if sugar, then energy)
• White, Jay D.  Research in Public Administration . Conn., 1998.
• Schick, Theodore, and Lewis Vaughn.  How to Think about Weird Things: Critical Thinking for a New Age . McGraw-Hill Higher Education, 2002.
• Null Hypothesis Examples
• Examples of Independent and Dependent Variables
• Difference Between Independent and Dependent Variables
• Null Hypothesis Definition and Examples
• Definition of a Hypothesis
• What Are the Elements of a Good Hypothesis?
• Six Steps of the Scientific Method
• Independent Variable Definition and Examples
• What Are Examples of a Hypothesis?
• Understanding Simple vs Controlled Experiments
• Scientific Method Flow Chart
• Scientific Method Vocabulary Terms
• What Is a Testable Hypothesis?
• What 'Fail to Reject' Means in a Hypothesis Test
• How To Design a Science Fair Experiment
• What Is an Experiment? Definition and Design

• Search Search Please fill out this field.

What Is Hypothesis Testing?

• How It Works

4 Step Process

The bottom line.

• Fundamental Analysis

Hypothesis Testing: 4 Steps and Example

Hypothesis testing, sometimes called significance testing, is an act in statistics whereby an analyst tests an assumption regarding a population parameter. The methodology employed by the analyst depends on the nature of the data used and the reason for the analysis.

Hypothesis testing is used to assess the plausibility of a hypothesis by using sample data. Such data may come from a larger population or a data-generating process. The word "population" will be used for both of these cases in the following descriptions.

Key Takeaways

• Hypothesis testing is used to assess the plausibility of a hypothesis by using sample data.
• The test provides evidence concerning the plausibility of the hypothesis, given the data.
• Statistical analysts test a hypothesis by measuring and examining a random sample of the population being analyzed.
• The four steps of hypothesis testing include stating the hypotheses, formulating an analysis plan, analyzing the sample data, and analyzing the result.

How Hypothesis Testing Works

In hypothesis testing, an  analyst  tests a statistical sample, intending to provide evidence on the plausibility of the null hypothesis. Statistical analysts measure and examine a random sample of the population being analyzed. All analysts use a random population sample to test two different hypotheses: the null hypothesis and the alternative hypothesis.

The null hypothesis is usually a hypothesis of equality between population parameters; e.g., a null hypothesis may state that the population mean return is equal to zero. The alternative hypothesis is effectively the opposite of a null hypothesis. Thus, they are mutually exclusive , and only one can be true. However, one of the two hypotheses will always be true.

The null hypothesis is a statement about a population parameter, such as the population mean, that is assumed to be true.

• State the hypotheses.
• Formulate an analysis plan, which outlines how the data will be evaluated.
• Carry out the plan and analyze the sample data.
• Analyze the results and either reject the null hypothesis, or state that the null hypothesis is plausible, given the data.

Example of Hypothesis Testing

If an individual wants to test that a penny has exactly a 50% chance of landing on heads, the null hypothesis would be that 50% is correct, and the alternative hypothesis would be that 50% is not correct. Mathematically, the null hypothesis is represented as Ho: P = 0.5. The alternative hypothesis is shown as "Ha" and is identical to the null hypothesis, except with the equal sign struck-through, meaning that it does not equal 50%.

A random sample of 100 coin flips is taken, and the null hypothesis is tested. If it is found that the 100 coin flips were distributed as 40 heads and 60 tails, the analyst would assume that a penny does not have a 50% chance of landing on heads and would reject the null hypothesis and accept the alternative hypothesis.

If there were 48 heads and 52 tails, then it is plausible that the coin could be fair and still produce such a result. In cases such as this where the null hypothesis is "accepted," the analyst states that the difference between the expected results (50 heads and 50 tails) and the observed results (48 heads and 52 tails) is "explainable by chance alone."

When Did Hypothesis Testing Begin?

Some statisticians attribute the first hypothesis tests to satirical writer John Arbuthnot in 1710, who studied male and female births in England after observing that in nearly every year, male births exceeded female births by a slight proportion. Arbuthnot calculated that the probability of this happening by chance was small, and therefore it was due to “divine providence.”

What are the Benefits of Hypothesis Testing?

Hypothesis testing helps assess the accuracy of new ideas or theories by testing them against data. This allows researchers to determine whether the evidence supports their hypothesis, helping to avoid false claims and conclusions. Hypothesis testing also provides a framework for decision-making based on data rather than personal opinions or biases. By relying on statistical analysis, hypothesis testing helps to reduce the effects of chance and confounding variables, providing a robust framework for making informed conclusions.

What are the Limitations of Hypothesis Testing?

Hypothesis testing relies exclusively on data and doesn’t provide a comprehensive understanding of the subject being studied. Additionally, the accuracy of the results depends on the quality of the available data and the statistical methods used. Inaccurate data or inappropriate hypothesis formulation may lead to incorrect conclusions or failed tests. Hypothesis testing can also lead to errors, such as analysts either accepting or rejecting a null hypothesis when they shouldn’t have. These errors may result in false conclusions or missed opportunities to identify significant patterns or relationships in the data.

Hypothesis testing refers to a statistical process that helps researchers determine the reliability of a study. By using a well-formulated hypothesis and set of statistical tests, individuals or businesses can make inferences about the population that they are studying and draw conclusions based on the data presented. All hypothesis testing methods have the same four-step process, which includes stating the hypotheses, formulating an analysis plan, analyzing the sample data, and analyzing the result.

Sage. " Introduction to Hypothesis Testing ," Page 4.

Elder Research. " Who Invented the Null Hypothesis? "

Formplus. " Hypothesis Testing: Definition, Uses, Limitations and Examples ."

• Terms of Service
• Editorial Policy
• Privacy Policy
• Your Privacy Choices

Hypothesis Testing

Hypothesis testing is a tool for making statistical inferences about the population data. It is an analysis tool that tests assumptions and determines how likely something is within a given standard of accuracy. Hypothesis testing provides a way to verify whether the results of an experiment are valid.

A null hypothesis and an alternative hypothesis are set up before performing the hypothesis testing. This helps to arrive at a conclusion regarding the sample obtained from the population. In this article, we will learn more about hypothesis testing, its types, steps to perform the testing, and associated examples.

What is Hypothesis Testing in Statistics?

Hypothesis testing uses sample data from the population to draw useful conclusions regarding the population probability distribution . It tests an assumption made about the data using different types of hypothesis testing methodologies. The hypothesis testing results in either rejecting or not rejecting the null hypothesis.

Hypothesis Testing Definition

Hypothesis testing can be defined as a statistical tool that is used to identify if the results of an experiment are meaningful or not. It involves setting up a null hypothesis and an alternative hypothesis. These two hypotheses will always be mutually exclusive. This means that if the null hypothesis is true then the alternative hypothesis is false and vice versa. An example of hypothesis testing is setting up a test to check if a new medicine works on a disease in a more efficient manner.

Null Hypothesis

The null hypothesis is a concise mathematical statement that is used to indicate that there is no difference between two possibilities. In other words, there is no difference between certain characteristics of data. This hypothesis assumes that the outcomes of an experiment are based on chance alone. It is denoted as \(H_{0}\). Hypothesis testing is used to conclude if the null hypothesis can be rejected or not. Suppose an experiment is conducted to check if girls are shorter than boys at the age of 5. The null hypothesis will say that they are the same height.

Alternative Hypothesis

The alternative hypothesis is an alternative to the null hypothesis. It is used to show that the observations of an experiment are due to some real effect. It indicates that there is a statistical significance between two possible outcomes and can be denoted as \(H_{1}\) or \(H_{a}\). For the above-mentioned example, the alternative hypothesis would be that girls are shorter than boys at the age of 5.

Hypothesis Testing P Value

In hypothesis testing, the p value is used to indicate whether the results obtained after conducting a test are statistically significant or not. It also indicates the probability of making an error in rejecting or not rejecting the null hypothesis.This value is always a number between 0 and 1. The p value is compared to an alpha level, \(\alpha\) or significance level. The alpha level can be defined as the acceptable risk of incorrectly rejecting the null hypothesis. The alpha level is usually chosen between 1% to 5%.

Hypothesis Testing Critical region

All sets of values that lead to rejecting the null hypothesis lie in the critical region. Furthermore, the value that separates the critical region from the non-critical region is known as the critical value.

Hypothesis Testing Formula

Depending upon the type of data available and the size, different types of hypothesis testing are used to determine whether the null hypothesis can be rejected or not. The hypothesis testing formula for some important test statistics are given below:

• z = \(\frac{\overline{x}-\mu}{\frac{\sigma}{\sqrt{n}}}\). \(\overline{x}\) is the sample mean, \(\mu\) is the population mean, \(\sigma\) is the population standard deviation and n is the size of the sample.
• t = \(\frac{\overline{x}-\mu}{\frac{s}{\sqrt{n}}}\). s is the sample standard deviation.
• \(\chi ^{2} = \sum \frac{(O_{i}-E_{i})^{2}}{E_{i}}\). \(O_{i}\) is the observed value and \(E_{i}\) is the expected value.

We will learn more about these test statistics in the upcoming section.

Types of Hypothesis Testing

Selecting the correct test for performing hypothesis testing can be confusing. These tests are used to determine a test statistic on the basis of which the null hypothesis can either be rejected or not rejected. Some of the important tests used for hypothesis testing are given below.

Hypothesis Testing Z Test

A z test is a way of hypothesis testing that is used for a large sample size (n ≥ 30). It is used to determine whether there is a difference between the population mean and the sample mean when the population standard deviation is known. It can also be used to compare the mean of two samples. It is used to compute the z test statistic. The formulas are given as follows:

• One sample: z = \(\frac{\overline{x}-\mu}{\frac{\sigma}{\sqrt{n}}}\).
• Two samples: z = \(\frac{(\overline{x_{1}}-\overline{x_{2}})-(\mu_{1}-\mu_{2})}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}}}}\).

Hypothesis Testing t Test

The t test is another method of hypothesis testing that is used for a small sample size (n < 30). It is also used to compare the sample mean and population mean. However, the population standard deviation is not known. Instead, the sample standard deviation is known. The mean of two samples can also be compared using the t test.

• One sample: t = \(\frac{\overline{x}-\mu}{\frac{s}{\sqrt{n}}}\).
• Two samples: t = \(\frac{(\overline{x_{1}}-\overline{x_{2}})-(\mu_{1}-\mu_{2})}{\sqrt{\frac{s_{1}^{2}}{n_{1}}+\frac{s_{2}^{2}}{n_{2}}}}\).

Hypothesis Testing Chi Square

The Chi square test is a hypothesis testing method that is used to check whether the variables in a population are independent or not. It is used when the test statistic is chi-squared distributed.

One Tailed Hypothesis Testing

One tailed hypothesis testing is done when the rejection region is only in one direction. It can also be known as directional hypothesis testing because the effects can be tested in one direction only. This type of testing is further classified into the right tailed test and left tailed test.

Right Tailed Hypothesis Testing

The right tail test is also known as the upper tail test. This test is used to check whether the population parameter is greater than some value. The null and alternative hypotheses for this test are given as follows:

\(H_{0}\): The population parameter is ≤ some value

\(H_{1}\): The population parameter is > some value.

If the test statistic has a greater value than the critical value then the null hypothesis is rejected

Left Tailed Hypothesis Testing

The left tail test is also known as the lower tail test. It is used to check whether the population parameter is less than some value. The hypotheses for this hypothesis testing can be written as follows:

\(H_{0}\): The population parameter is ≥ some value

\(H_{1}\): The population parameter is < some value.

The null hypothesis is rejected if the test statistic has a value lesser than the critical value.

Two Tailed Hypothesis Testing

In this hypothesis testing method, the critical region lies on both sides of the sampling distribution. It is also known as a non - directional hypothesis testing method. The two-tailed test is used when it needs to be determined if the population parameter is assumed to be different than some value. The hypotheses can be set up as follows:

\(H_{0}\): the population parameter = some value

\(H_{1}\): the population parameter ≠ some value

The null hypothesis is rejected if the test statistic has a value that is not equal to the critical value.

Hypothesis Testing Steps

Hypothesis testing can be easily performed in five simple steps. The most important step is to correctly set up the hypotheses and identify the right method for hypothesis testing. The basic steps to perform hypothesis testing are as follows:

• Step 1: Set up the null hypothesis by correctly identifying whether it is the left-tailed, right-tailed, or two-tailed hypothesis testing.
• Step 2: Set up the alternative hypothesis.
• Step 3: Choose the correct significance level, \(\alpha\), and find the critical value.
• Step 4: Calculate the correct test statistic (z, t or \(\chi\)) and p-value.
• Step 5: Compare the test statistic with the critical value or compare the p-value with \(\alpha\) to arrive at a conclusion. In other words, decide if the null hypothesis is to be rejected or not.

Hypothesis Testing Example

The best way to solve a problem on hypothesis testing is by applying the 5 steps mentioned in the previous section. Suppose a researcher claims that the mean average weight of men is greater than 100kgs with a standard deviation of 15kgs. 30 men are chosen with an average weight of 112.5 Kgs. Using hypothesis testing, check if there is enough evidence to support the researcher's claim. The confidence interval is given as 95%.

Step 1: This is an example of a right-tailed test. Set up the null hypothesis as \(H_{0}\): \(\mu\) = 100.

Step 2: The alternative hypothesis is given by \(H_{1}\): \(\mu\) > 100.

Step 3: As this is a one-tailed test, \(\alpha\) = 100% - 95% = 5%. This can be used to determine the critical value.

1 - \(\alpha\) = 1 - 0.05 = 0.95

0.95 gives the required area under the curve. Now using a normal distribution table, the area 0.95 is at z = 1.645. A similar process can be followed for a t-test. The only additional requirement is to calculate the degrees of freedom given by n - 1.

Step 4: Calculate the z test statistic. This is because the sample size is 30. Furthermore, the sample and population means are known along with the standard deviation.

z = \(\frac{\overline{x}-\mu}{\frac{\sigma}{\sqrt{n}}}\).

\(\mu\) = 100, \(\overline{x}\) = 112.5, n = 30, \(\sigma\) = 15

z = \(\frac{112.5-100}{\frac{15}{\sqrt{30}}}\) = 4.56

Step 5: Conclusion. As 4.56 > 1.645 thus, the null hypothesis can be rejected.

Hypothesis Testing and Confidence Intervals

Confidence intervals form an important part of hypothesis testing. This is because the alpha level can be determined from a given confidence interval. Suppose a confidence interval is given as 95%. Subtract the confidence interval from 100%. This gives 100 - 95 = 5% or 0.05. This is the alpha value of a one-tailed hypothesis testing. To obtain the alpha value for a two-tailed hypothesis testing, divide this value by 2. This gives 0.05 / 2 = 0.025.

Related Articles:

• Probability and Statistics
• Data Handling

Important Notes on Hypothesis Testing

• Hypothesis testing is a technique that is used to verify whether the results of an experiment are statistically significant.
• It involves the setting up of a null hypothesis and an alternate hypothesis.
• There are three types of tests that can be conducted under hypothesis testing - z test, t test, and chi square test.
• Hypothesis testing can be classified as right tail, left tail, and two tail tests.

Examples on Hypothesis Testing

• Example 1: The average weight of a dumbbell in a gym is 90lbs. However, a physical trainer believes that the average weight might be higher. A random sample of 5 dumbbells with an average weight of 110lbs and a standard deviation of 18lbs. Using hypothesis testing check if the physical trainer's claim can be supported for a 95% confidence level. Solution: As the sample size is lesser than 30, the t-test is used. \(H_{0}\): \(\mu\) = 90, \(H_{1}\): \(\mu\) > 90 \(\overline{x}\) = 110, \(\mu\) = 90, n = 5, s = 18. \(\alpha\) = 0.05 Using the t-distribution table, the critical value is 2.132 t = \(\frac{\overline{x}-\mu}{\frac{s}{\sqrt{n}}}\) t = 2.484 As 2.484 > 2.132, the null hypothesis is rejected. Answer: The average weight of the dumbbells may be greater than 90lbs
• Example 2: The average score on a test is 80 with a standard deviation of 10. With a new teaching curriculum introduced it is believed that this score will change. On random testing, the score of 38 students, the mean was found to be 88. With a 0.05 significance level, is there any evidence to support this claim? Solution: This is an example of two-tail hypothesis testing. The z test will be used. \(H_{0}\): \(\mu\) = 80, \(H_{1}\): \(\mu\) ≠ 80 \(\overline{x}\) = 88, \(\mu\) = 80, n = 36, \(\sigma\) = 10. \(\alpha\) = 0.05 / 2 = 0.025 The critical value using the normal distribution table is 1.96 z = \(\frac{\overline{x}-\mu}{\frac{\sigma}{\sqrt{n}}}\) z = \(\frac{88-80}{\frac{10}{\sqrt{36}}}\) = 4.8 As 4.8 > 1.96, the null hypothesis is rejected. Answer: There is a difference in the scores after the new curriculum was introduced.
• Example 3: The average score of a class is 90. However, a teacher believes that the average score might be lower. The scores of 6 students were randomly measured. The mean was 82 with a standard deviation of 18. With a 0.05 significance level use hypothesis testing to check if this claim is true. Solution: The t test will be used. \(H_{0}\): \(\mu\) = 90, \(H_{1}\): \(\mu\) < 90 \(\overline{x}\) = 110, \(\mu\) = 90, n = 6, s = 18 The critical value from the t table is -2.015 t = \(\frac{\overline{x}-\mu}{\frac{s}{\sqrt{n}}}\) t = \(\frac{82-90}{\frac{18}{\sqrt{6}}}\) t = -1.088 As -1.088 > -2.015, we fail to reject the null hypothesis. Answer: There is not enough evidence to support the claim.

go to slide go to slide go to slide

Book a Free Trial Class

FAQs on Hypothesis Testing

What is hypothesis testing.

Hypothesis testing in statistics is a tool that is used to make inferences about the population data. It is also used to check if the results of an experiment are valid.

What is the z Test in Hypothesis Testing?

The z test in hypothesis testing is used to find the z test statistic for normally distributed data . The z test is used when the standard deviation of the population is known and the sample size is greater than or equal to 30.

What is the t Test in Hypothesis Testing?

The t test in hypothesis testing is used when the data follows a student t distribution . It is used when the sample size is less than 30 and standard deviation of the population is not known.

What is the formula for z test in Hypothesis Testing?

The formula for a one sample z test in hypothesis testing is z = \(\frac{\overline{x}-\mu}{\frac{\sigma}{\sqrt{n}}}\) and for two samples is z = \(\frac{(\overline{x_{1}}-\overline{x_{2}})-(\mu_{1}-\mu_{2})}{\sqrt{\frac{\sigma_{1}^{2}}{n_{1}}+\frac{\sigma_{2}^{2}}{n_{2}}}}\).

What is the p Value in Hypothesis Testing?

The p value helps to determine if the test results are statistically significant or not. In hypothesis testing, the null hypothesis can either be rejected or not rejected based on the comparison between the p value and the alpha level.

What is One Tail Hypothesis Testing?

When the rejection region is only on one side of the distribution curve then it is known as one tail hypothesis testing. The right tail test and the left tail test are two types of directional hypothesis testing.

What is the Alpha Level in Two Tail Hypothesis Testing?

To get the alpha level in a two tail hypothesis testing divide \(\alpha\) by 2. This is done as there are two rejection regions in the curve.

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons

Margin Size

• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

6.1: Random Effects

• Last updated
• Save as PDF
• Page ID 33658

• Penn State's Department of Statistics
• The Pennsylvania State University

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

\( \newcommand{\Span}{\mathrm{span}}\)

\( \newcommand{\id}{\mathrm{id}}\)

\( \newcommand{\kernel}{\mathrm{null}\,}\)

\( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\)

\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\)

\( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

\( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vectorC}[1]{\textbf{#1}} \)

\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

When a treatment (or factor) is a random effect, the model specifications together with relevant null and alternative hypotheses will have to be changed. Recall the cell means model defined in Chapter 4 for the fixed effect case, which has the model equation: \[Y_{ij} = \mu_{i} + \epsilon_{ij}\] where \(\mu_{i}\) are parameters for the treatment means.

For the single factor random effects model we have: \[Y_{ij} = \mu_{i} + \epsilon_{ij}\] where \(\mu_{i}\) and \(\epsilon_{ij}\) are independent random variables such that \(\mu_{i} \overset{iid}{\sim} \mathcal{N} \left(\mu, \sigma_{\mu}^{2}\right)\) and \(\epsilon_{ij} \overset{iid}{\sim} \mathcal{N} \left(0, \sigma_{\epsilon}^{2}\right)\). Here, \(i = 1, 2, \ldots, T\) and \(j = 1, 2, \ldots, n_{i}\), where \(n_{i} \equiv n\) if balanced.

Notice that the random effects ANOVA model is similar in appearance to the fixed effects ANOVA model. However, the treatment mean \(\mu_{i}\)'s are constant in the fixed-effect ANOVA model, whereas in the random-effects ANOVA model the treatment mean \(\mu_{i}\)'s are random variables.

Note that the expected mean response, in the random effects model stated above, is the same at every treatment level and equals \(\mu\).

\[E \left(Y_{ij}\right) = E \left(\mu_{i} + \epsilon_{ij}\right) = E \left(\mu_{i}\right) + E \left(\epsilon_{ij}\right) = \mu\]

The variance of the response variable (say \(\sigma_{Y}^{2}\)) in this case can be partitioned as: \[\sigma_{Y}^{2} = V \left(Y_{ij}\right) = V \left(\mu_{i} + \epsilon_{ij}\right) = V \left(\mu_{i}\right) + V \left(\epsilon_{ij}\right) = \sigma_{\mu}^{2} + \sigma_{\epsilon}^{2}\] as \(\mu_{i}\) and \(\epsilon_{ij}\) are independent random variables.

Similar to fixed effects ANOVA model, we can express the random effects ANOVA model using the factor effect representation, using \(\tau_{i} = \mu_{i} - \mu\). Therefore the factor effects representation of the random effects ANOVA model would be: \[Y_{ij} = \mu + \tau_{i} + \epsilon_{ij}\] where \(\mu\) is a constant overall mean, and \(\tau_{i}\) and \(\epsilon_{ij}\) are independent random variables such that \(\tau_{i} \overset{iid}{\sim} \mathcal{N} \left(0, \sigma_{\mu}^{2}\right)\) and \(\epsilon_{ij} \overset{iid}{\sim} \mathcal{N} \left(0, \sigma_{\epsilon}^{2}\right)\). Here, \(i = 1, 2, \ldots, T\) and \(j = 1, 2, \ldots, n_{i}\), where \(n_{i} \equiv n\) if balanced. Here, \(\tau_{i}\) is the effect of the randomly selected \(i^{th}\) level.

The terms \(\sigma_{\mu}^{2}\) and \(\sigma_{\epsilon}^{2}\) are referred to as variance components . In general, as will be seen later in more complex models, there will be a variance component associated with each effect involving at least one random factor.

Variance components play an important role in analyzing random effects data. They can be used to verify the significant contribution of each random effect to the variability of the response. For the single factor random-effects model stated above, the appropriate null and alternative hypothesis for this purpose is: \[H_{0}: \ \sigma_{\mu}^{2} = 0 \text{ vs. } H_{A}: \ \sigma_{\mu}^{2} > 0\]

Similar to the fixed effects model, an ANOVA analysis can then be carried out to determine if \(H_{0}\) can be rejected.

The MS and the df computations of the ANOVA table are the same for both the fixed and random-effects models. However, the computations of the F-statistics needed for hypothesis testing require some modification.

Specifically, the F statistics denominator will no longer always be the mean squared error (MSE or MSERROR) and will vary according to the effect of interest (listed in the Source column of the ANOVA table). For a random-effects model, the quantities known as Expected Means Squares (EMS) , shown in the ANOVA table below , can be used to identify the appropriate F-statistic denominator for a given source in the ANOVA table. These EMS quantities will also be useful in estimating the variance components associated with a given random effect. Note that the EMS quantities are in fact the population counterparts of the mean sums of squares (MS) that we are already familiar with. In SAS the proc mixed , method=type3 option will generate the EMS column in the ANOVA table output.

Variance components are NOT synonymous with mean sums of squares. Variance components are usually estimated by using the Method of Moments where algebraic equations, created by setting the mean sums of squares (MS) equal to the EMS for the relevant effects, are solved for the unknown variance components. For example, the variance component for the treatment in the single-factor random effects discussed above can be solved as:

\[s_{\text{among trts}}^{2} = \frac{MS_{trt} - MS_{error}}{n}\]

This is by using the two equations:

\(MS_{error} = \sigma_{\epsilon}^{2}\)

\(MS_{trt} = \sigma_{\epsilon}^{2} + n \sigma_{\mu}^{2}\)

More about variance components...

Often the variance component of a specific effect in the model is expressed as a percent of the total variation of the variation in the response variable.

Another common application of variance components is when researchers are interested in the relative size of the treatment effect compared to the within-treatment level variation. This leads to a quantity called the intraclass correlation coefficient (ICC), defined as: \[ICC = \frac{\sigma_{\text{among trts}}^{2}}{\sigma_{\text{\text{among trts}}^{2} + \sigma_{\text{within trts}}^{2}}\]

For single random factor studies, \(ICC = \frac{\sigma_{mu}^{2}}{\sigma_{mu}^{2} + \sigma_{\epsilon}^{2}}\). ICC can also be thought of as the correlation between the observations within the group (i.e. \(\text{corr} \left(Y_{ij}, Y_{ij'}\right)\), where \(j \neq j'\). Small values of ICC indicate a large spread of values at each level of the treatment, whereas large values of ICC indicate relatively little spread at each level of the treatment:

Loading metrics

Open Access

Peer-reviewed

Research Article

Interpretable online network dictionary learning for inferring long-range chromatin interactions

Roles Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliation Department of Electrical and Computer Engineering, University of Illinois, Urbana-Champaign, Illinois, United States of America

Roles Formal analysis, Investigation, Methodology, Writing – review & editing

Affiliation Department of Mathematics, University of Wisconsin - Madison, Madison, Wisconsin, United States of America

Roles Data curation, Writing – review & editing

Affiliation School of Biological and Health Systems Engineering, Arizona State University, Phoenix, Arizona, United States of America

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliation Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, Michigan, United States of America

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

• Vishal Rana, 
• Jianhao Peng, 
• Chao Pan, 
• Hanbaek Lyu, 
• Albert Cheng, 
• Minji Kim, 
• Olgica Milenkovic

• Published: May 16, 2024
• https://doi.org/10.1371/journal.pcbi.1012095
• Reader Comments

This is an uncorrected proof.

Dictionary learning (DL), implemented via matrix factorization (MF), is commonly used in computational biology to tackle ubiquitous clustering problems. The method is favored due to its conceptual simplicity and relatively low computational complexity. However, DL algorithms produce results that lack interpretability in terms of real biological data. Additionally, they are not optimized for graph-structured data and hence often fail to handle them in a scalable manner.

In order to address these limitations, we propose a novel DL algorithm called online convex network dictionary learning (online cvxNDL). Unlike classical DL algorithms, online cvxNDL is implemented via MF and designed to handle extremely large datasets by virtue of its online nature. Importantly, it enables the interpretation of dictionary elements, which serve as cluster representatives, through convex combinations of real measurements. Moreover, the algorithm can be applied to data with a network structure by incorporating specialized subnetwork sampling techniques.

To demonstrate the utility of our approach, we apply cvxNDL on 3D-genome RNAPII ChIA-Drop data with the goal of identifying important long-range interaction patterns (long-range dictionary elements). ChIA-Drop probes higher-order interactions, and produces data in the form of hypergraphs whose nodes represent genomic fragments. The hyperedges represent observed physical contacts. Our hypergraph model analysis has the objective of creating an interpretable dictionary of long-range interaction patterns that accurately represent global chromatin physical contact maps. Through the use of dictionary information, one can also associate the contact maps with RNA transcripts and infer cellular functions.

To accomplish the task at hand, we focus on RNAPII-enriched ChIA-Drop data from Drosophila Melanogaster S2 cell lines. Our results offer two key insights. First, we demonstrate that online cvxNDL retains the accuracy of classical DL (MF) methods while simultaneously ensuring unique interpretability and scalability. Second, we identify distinct collections of proximal and distal interaction patterns involving chromatin elements shared by related processes across different chromosomes, as well as patterns unique to specific chromosomes. To associate the dictionary elements with biological properties of the corresponding chromatin regions, we employ Gene Ontology (GO) enrichment analysis and perform multiple RNA coexpression studies.

Author summary

We introduce a novel method for dictionary learning termed online convex Network Dictionary Learning (online cvxNDL). The method operates in an online manner and utilizes representative subnetworks of a network dataset as dictionary elements. A key feature of online cvxNDL is its ability to work with graph-structured data and generate dictionary elements that represent convex combinations of real data points, thus ensuring interpretability.

Online cvxNDL is used to investigate long-range chromatin interactions in S2 cell lines of Drosophila Melanogaster obtained through RNAPII ChIA-Drop measurements represented as hypergraphs. The results show that dictionary elements can accurately and efficiently reconstruct the original interactions present in the data, even when subjected to convexity constraints. To shed light on the biological relevance of the identified dictionaries, we perform Gene Ontology enrichment and RNA-seq coexpression analyses. These studies uncover multiple long-range interaction patterns that are chromosome-specific. Furthermore, the findings affirm the significance of convex dictionaries in representing TADs cross-validated by imaging methods (such as 3-color FISH (fluorescence in situ hybridization)).

Citation: Rana V, Peng J, Pan C, Lyu H, Cheng A, Kim M, et al. (2024) Interpretable online network dictionary learning for inferring long-range chromatin interactions. PLoS Comput Biol 20(5): e1012095. https://doi.org/10.1371/journal.pcbi.1012095

Editor: Tamar Schlick, New York University, UNITED STATES

Received: December 16, 2023; Accepted: April 20, 2024; Published: May 16, 2024

Copyright: © 2024 Rana et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The code is available at https://github.com/rana95vishal/chromatin_DL The complete dataset is available at https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE109355 .

Funding: The work was supported by the National Science Foundation grants #1956384 (AC, MK, and OM), #2206296 (OM) and grant CZI DAF 2022-249217 (OM and MK). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Dictionary learning (DL) is a widely used method in learning and computational biology for approximating a matrix through sparse linear combinations of dictionary elements. DL has been used in various applications such as clustering, denoising, data compression, and extracting low-dimensional patterns [ 1 – 8 ]. For example, DL is used to cluster data points since dictionary elements essentially represent centroids of clusters. DL can perform denoising by combining only the highest-score dictionary elements to reconstruct the input; in this case, the low-score dictionary elements reflect the distortion in the data due to noise. DL can also perform efficient data compression by storing only the dictionary elements and associated weights needed for reconstruction. In addition, DL can be used to extract low-dimensional patterns from complex high-dimensional inputs.

However, standard DL methods [ 9 , 10 ] suffer from interpretability and scalability issues and are primarily applied to unstructured data. To address interpretability issues for unstructured data, convex matrix factorization was introduced in [ 11 ]. Convex matrix factorization requires that the dictionary elements be convex combinations of real data points, thereby introducing a constraint that adds to the computational complexity of the method. At the same time, to improve scalability, DL and convex DL algorithms can be adapted to online settings [ 12 , 13 ]. Network DL (NDL), introduced in [ 14 ], operates on graph-structured data and samples subnetworks via Markov Chain Monte Carlo (MCMC) methods [ 14 – 16 ] to efficiently and accurately identify a small number of subnetwork dictionary elements that best explain subgraph-level interactions of the entire global network. These dictionary elements learned by the original NDL algorithm only provide ‘latent’ subgraph structures that are not necessarily associated with specific subgraphs in the network. When applied to gene interaction networks, such latent subnetworks cannot be associated with specific genomic regions or viewed as physical interactions between genomic loci, making the method biologically uninterpretable.

To address the shortcoming of online NDL, we propose online cvxNDL, a novel NDL method that combines the MCMC sampling technique from [ 14 ] with convexity constraints on the matrix representation of sampled subnetworks. These constraints are handled through the concept of “dictionary element representatives,” which are essentially adjacency matrices of real subnetworks of the input network. The representatives are used as building blocks of actual dictionary elements. More precisely, dictionary elements are convex combinations of small subsets of representatives. This allows us to map the dictionary element entries to actual genomic regions and view them as real physical interactions. The online learning component is handled via sequential updates of the best choice of representative elements, complementing the approach proposed in [ 13 ] for unstructured data. This formulation ensures interpretability of the results and allows for scaling to large datasets.

The utility of online cvxNDL is demonstrated by performing an extensive analysis of 3D chromatin interaction data generated by the RNAPII ChIA-Drop [ 17 ] technique. Chromatin 3D structures play a crucial role in gene regulation [ 18 , 19 ] and have traditionally been measured using “bulk” sequencing methods, such as Hi-C [ 20 ] and ChIA-PET [ 21 , 22 ]. However, due to the proximity ligation step, these methods can only capture pairwise contacts and fail to extract potential multiway interactions that exist in the cell. Further, these methods operate on a population of millions of molecules and therefore only provide information about population averages. ChIA-Drop, by contrast, mitigates these issues by employing droplet-based barcode-linked sequencing to capture multiway chromatin interactions at the single-molecule level, enabling the detection of short- and long-range interactions involving multiple genomic loci. Note that, more specifically, RNAPII ChIA-Drop data elucidates interactions among regulatory elements such as enhancers and promoters, which warrants contrasting/combining it with RNA-seq data.

The cvxNDL method is first tested on synthetic data, and, subsequently, on real-world RNAPII ChIA-Drop data pertaining to chromosomes of Drosophila Melanogaster Schneider 2 (S2) phagocytic cell lines (Due to the limited number of complete ChIA-Drop datasets, we only report findings for cell-lines also studied in [ 17 ]). For simplicity, we henceforth refer to the latter as ChIA-Drop data (Our method is designed to handle multiway interactions generated by ChIA-Drop experiments and to generate dictionary elements that capture fundamental chromatin interactions. However, it can also be directly applied to other conformation maps, including Hi-C matrices, but without the hypergraph preprocessing steps). Our findings are multi-fold.

First, we provide dictionary elements that can be used to represent chromatin interactions in a succinct and highly accurate manner.

Second, we discover significant differences between the long-range interactions captured by dictionary elements of different chromosomes. These differences can also be summarized via the average distance between interacting genomic loci and the densities of interactions.

Third, we perform Gene Ontology (GO) enrichment analysis to gain insights into the collective functionality of the genomic regions represented by the dictionary elements of different chromosomes. As an example, for chromosomes 2L and 2R, our GO enrichment analysis reveals significant enrichment in several important terms related to reproduction, oocyte differentiation, and embryonic development. Likewise, chromosomes 3L and 3R are enriched in key GO terms associated with blood circulation and response to heat and cold.

Fourth, to further validate the utility of the dictionary elements, we perform an RNA-Seq coexpression analysis using data from independent experiments conducted on Drosophila Melanogaster S2 cell lines, available through the NCBI Sequence Read Archive [ 23 ]. We show that genes associated with a given dictionary element exhibit high levels of coexpression, as validated on TAD interactions T1-T4 and R1-R4 [ 17 ]. Notably, a small subset of our dictionary elements is able to accurately represent these TAD regions and their multiway interactions, confirming the capability of our method to effectively capture complex patterns of both short- and long-range interactions. In addition, we map our dictionary elements onto interaction networks, including the STRING protein-protein interaction network [ 24 ], as well as large gene expression repositories like FlyMine. We observe closely coordinated coexpression among the identified genes, further supporting the biological relevance of the identified dictionary elements.

With its unique features, our new interpretable method for dictionary learning adds to the growing literature on machine learning approaches that aim to elucidate properties of chromatin interactions [ 25 – 28 ].

Results and discussion

We first provide an intuitive, high-level overview of the steps of the interpretable dictionary learning method, as illustrated in Fig 1 . The figure describes the most important global ideas behind our novel online cvxNDL pipeline. A rigorous mathematical formulation of the problem and relevant analyses are delegated to the Methods Section, while detailed algorithmic methods are available in Section B in S1 Text .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pcbi.1012095.g001

Chromatin interactions are commonly represented as contact maps. A contact map can be viewed as a hypergraph, where nodes represent genomic loci and two or more such nodes are connected through hyperedges to represent experimentally observed multiway chromatin interactions. Since it is challenging to work with hypergraphs directly, the first step is to transform a hypergraph into an ordinary network (graph), which we tacitly assume is connected. For this purpose, we employ clique expansion [ 29 , 30 ], as shown in Fig 1B . Clique expansion converts a hyperedge into a clique (a fully connected network) and therefore preserves all interactions encapsulated by the hyperedge. However, large hyperedges covering roughly 10 or more nodes in the network can introduce distortion by creating new cliques that do not correspond to any multiway interaction, as shown in Fig 1C [ 31 ]. The frequency of such large hyperedges and the total number of hyperedges in chromatin interaction data is limited (i.e., the hypergraph is sparse, see Table A in S1 Text ). This renders the distortion due to the hypergraph-to-network conversion process negligible.

To generate an online sample from the clique-expanded input network, we use a subnetwork sampling procedure shown in Fig 1D . We consider a small template network consisting of a fixed number of nodes and search for induced subnetworks in the input that contain the template network topology. These induced subnetworks can be rigorously characterized via homomorphisms and are discussed in detail in the Methods Section. An example of a homomorphism is shown in Fig 1D . Throughout our analysis, we will exclusively focus on path homomorphisms because they are most suitable for the biological problem investigated. To generate a sequence of online samples from the input network, we employ MCMC sampling. Given a path sample at discrete time t , the next sample at time t + 1 is generated by selecting a new node uniformly at random from the neighborhood of the sample at time t and calculating its probability of acceptance β , explained in the Methods Section. If this new node is accepted, we perform a directed random walk starting at the selected node, otherwise, we restart the random walk from the first node of the sample at time t . Note that the input network is undirected while only the sampling method requires a directed walk as the order of the labeled nodes matters. (see Fig 1E ).

MCMC sampling is used to generate a sequence of samples to initialize a dictionary with K dictionary elements , where K is chosen based on the properties of the dataset. Each of the dictionary elements is represented as a convex combination of a small (sparse) set of representatives that are real biological observations. The convex hull of these representatives is termed the representative region of the dictionary element. As a result, the vertices of the representative regions comprise a collection of MCMC-generated real-world samples. Fig 2A shows the organization of a dictionary as a collection of dictionary elements, representatives, and representative regions.

After initialization, we perform iterative optimization of the DL objective function using online samples, again generated via the MCMC method. More precisely, at each iteration, we compute the distance between the new sample and every current estimate of dictionary elements. Subsequently, we assign the sample to the representative region of the nearest dictionary element, which leads to an increase in the size of the set of representatives associated with the dictionary element. From this expanded set of representatives, we carefully select one representative for removal, maximizing the improvement in the quality of our dictionary element and the objective function. It is possible that the removed representative is the newly added data sample assigned to the representative region. In this case, the dictionary element remains unchanged. Otherwise, it is obtained as a convex combination of the updated set of representatives. After observing sufficiently many online samples, the algorithm converges to an accurate set of dictionary elements or the procedure terminates without convergence (in which case we declare a failure and restart the learning process). In our experiments, we never terminated with failure, but due to the lack of provable convergence guarantees for real-world datasets, such scenarios cannot be precluded. The update procedure is shown in Fig 2B .

(a) Organization of a dictionary comprising K dictionary elements that are convex combinations of real representative subnetworks. Each dictionary element itself is a sparse convex combination of a set of representatives which are small subnetworks of the input real-world network. In the example, there are 6 options for the representatives, and inclusion of a representative into a dictionary element is indicated by a colored entry in a 6-dimensional indicator column-vector. Each of the 6 representatives corresponds to a subnetwork of the input network with a fixed number of nodes (3 for our example). The dictionary element is generated by a convex combination of the corresponding adjacency matrices of its corresponding representative subnetworks. For the example, the resulting dictionary elements are 9 × 9 matrices. (b) Illustration of the representative region update. When an online data sample is observed, the distance of the sample to each of the current dictionary elements is computed and the sample is assigned to the representative region of the nearest dictionary element. From this expanded set of representatives, one representative is carefully selected for removal to improve the objective. The new dictionary element is then obtained as an optimized convex combination of the updated set of representatives.

https://doi.org/10.1371/journal.pcbi.1012095.g002

We applied the method outlined above to RNAPII-enriched ChIA-Drop data from Drosophila Melanogaster S2 cells, using a dm3 reference genome [ 17 ], to learn dictionaries of chromatin interactions. Fig 3 provides an illustration of the ChIA-Drop pipeline.

ChIA-Drop [ 17 ] adopts a droplet-based barcode-linked technique to reveal multiway chromatin interactions at a single molecule level. Chromatin samples are crosslinked and fragmented without a proximity ligation step. The samples are enriched for informative fragments through antibody pull-down.

https://doi.org/10.1371/journal.pcbi.1012095.g003

We preprocessed the RNAPII ChIA-Drop data to remove fragments mapped to the repetitive regions in the genome and performed an MIA-Sig enrichment test with FDR 0.1 [ 32 ]. Only the hyperedges that passed this test were used in subsequent analysis. The highest interaction resolution of our method is dictated by the technology used to generate the data. Since ChIA-Drop experiments involved genomic fragments of length ≤ 630 bases [ 17 ], we binned chromosomal genetic sequences into fragments of 500 bases each and used the midpoint of each fragment for distance evaluations and dictionary element mappings onto chromatin order. These bins of 500 consecutive bases form the nodes of the hypergraph for each chromosome, while the set of filtered multiway interactions form the hyperedges. The dataset hence includes 45, 938, 42, 292, 49, 072, and 55, 795 nodes and 36, 140, 28, 387, 53, 006, 45, 530 hyperedges for chromosome chr2L, chr2R, chr3L and chr3R respectively. The distribution of the hyperedge sizes is given in Table A in S1 Text . To create networks from hypergraphs, we converted the multiway interactions into cliques. The clique-expanded input network has 113, 606, 85, 316, 161, 590, and 143, 370 edges respectively. Although the ChIA-Drop data comprises interactions from six chromosomes chr2L, chr2R, chr3L, chr3R, chr4 and chrX, since chr4 and chrX are relatively short regions and most of the functional genes are located on chr2L, chr2R, chr3L, and chr3R, we focus our experiments only on the latter.

In the analyses, we fix the number of dictionary elements to K = 25. Clearly, other genomic datasets may benefit from a different choice of the parameter K , which has to be fine-tuned for each different dataset. Also, as template subnetworks, we use paths , since paths are the simplest and most common network motifs, especially in chromatin interaction data (most contact measurements are proximal due to the linear chromosome order). We select paths of length 21 nodes (i.e., 21 × 500 bases). Once again, both the choice of the subnetwork (motif) and its number of constituent nodes is data dependent. The detailed explanation below justifies our parameter choices for the Drosophila dataset.

The typical range of long-range interactions in chromatin structures depends on the species/reference genome. For Drosophila Melanogaster , TADs are 10, 000–100, 000 bases long, while loops are usually (much) shorter than 10, 000 bases [ 17 , 33 ]. This suggests using 10, 000 bases as an approximate lower bound for the length of long-range interactions. In addition, within the network itself, the size of the genomic bins dictates what path lengths correspond to long-range interactions. This influences the length of sampling motifs chosen for the MCMC sampling step—the sampled paths should be long enough to capture long-range interactions. Paths of length 21 nodes result in 21 × 500 = 10, 500 bases in the chromosome, which in turn amounts to a length of approximately 10, 000 bases. Additionally, the choice for the path-length also controls the trade-off between the number of representatives and their size. With a choice of path-length as above, we have to draw 20, 000 MCMC samples to cover all the nodes (chromatin fragments) in the dataset. This is evidenced by Fig D in S1 Text which plots the number of MCMC samples needed for given percentages of node coverage.

Similarly, the choice for K , the number of dictionary elements used, also depends on the dataset. Promoters and enhancers only constitute a very small fraction of the entire length of noncoding DNA. Studies indicate the existence of 10, 000 to 12, 000 such regions in the Drosophila genome, with each region being 100–1000 bases in length [ 34 , 35 ]. Working with the upper range of values, we arrive at a total length upper bounded by 12, 000 × 1, 000 = 12, 000, 000 bases for the promoter/enhancer regions (one should compare this to the total length of the genome, which equals 180, 000, 000 bases). With K = 25 for each of the 4 chromosomes, the dictionary elements will cover approximately 4 × 25 × 10 × 10, 000 = 10, 000, 000 bases which is close to the (loose) upper-bound estimate for the total length of the promoter/enhancer regions.

As a final remark, performing a multidimensional grid search for hyperparameters may be computationally prohibitive. Also, the procedure outlined above relies on solid biological side-information.

MCMC sampling for initialization, as well as for subsequent online optimization steps, was performed before running the online optimization process to improve the efficiency of our implementation. We sampled 20, 000 subnetworks from each of the four chromosomes to ensure sufficient coverage of the input network. From this pool of subnetworks, we randomly selected 500 subnetworks to initialize our dictionaries, ensuring that each dictionary element had at least 10 representatives (which suffice to get quality initializations for the dictionary elements themselves). Each online step involved sampling an additional subnetwork and we iterated this procedure up to 1 million times, as needed for convergence (see Fig 1A ).

At this point, it is crucial to observe that the dictionary elements learned by online cvxNDL effectively capture long-range interactions because each dictionary element may include distal genomic regions that are not adjacent in the genomic order. In other words, the diagonal entries of our dictionary elements do not exclusively represent consecutive genomic regions as in standard chromatin contact maps; instead, they may include both nonconsecutive (long-range) and consecutive (short-range, adjacent) interactions. This point is explained in detail in Fig 4 . Another relevant remark is that without the convexity constraint, dictionary element entries could not have been meaningfully mapped back (associated) to genomic regions and viewed as real physical interactions between genomic loci .

The elements on the diagonal are not necessarily indexed by adjacent (consecutive) genomic fragments, as explained by the example in the second row. There, off-diagonal long-range interactions in the 9 × 9 matrix are included in a 3 × 3 dictionary element whose diagonal elements are not in consecutive order.

https://doi.org/10.1371/journal.pcbi.1012095.g004

The dictionary elements generated from the Drosophila ChIA-Drop data for chr2L, chr2R, chr3L, and chr3R using the online cvxNDL method are shown in Fig 5 . Each subplot corresponds to one chromosome and has 25 dictionary elements ordered with respect to their importance scores , capturing the relevance and frequency of use of the dictionary element, and formally defined in the Methods Section. Each element is color-coded based on the genomic locations covered by their representatives. Hence, dictionary elements represent combinations of experimentally observed interaction patterns, uniquely capturing the significance of the genomic locations involved in the corresponding interactions. We also report the density and median distance between all consecutive pairs of interacting loci (connected nodes) of all dictionary elements in Tables B and C in S1 Text .

Each subplot contains 25 dictionary elements for the corresponding chromosome and each block in the subplots corresponds to one dictionary element. The elements are ordered by their importance score. Note that the “diagonals” in the dictionary elements do not exclusively represent localized topologically associated domains (TADs) as in standard chromatin contact maps; instead, they can also capture long-range interactions. This is due to the fact that the indices of the dictionary element matrices represent genomic regions that may be far apart in the genome. In contrast, standard contact maps have indices that correspond to continuously ordered genomic regions, so that the diagonals truly represent TADs (see Fig 4 ). The color-code captures the actual locations of the genomic regions involved in the representatives and their dictionary elements. The most interesting dictionary elements are those that contain both dark blue, light blue/green, and red colors (since they involve long-range interactions). This is especially the case for chr3L and chr3R.

https://doi.org/10.1371/journal.pcbi.1012095.g005

Note that our algorithm is the first method for online learning of convex (interpretable) network dictionaries. We can therefore only compare its representation accuracy to that of nonnegative matrix factorization (NMF), convex matrix factorization (CMF), and online network dictionary learning (online NDL). A comparison of the dictionaries formed through online cvxNDL and the aforementioned methods for chr2L is provided in Fig 6 .

NMF and CMF are learned off-line, using a total of 20, 000 samples. Note that these algorithms do not scale and cannot work with larger number of samples such as those used in online cvxNDL. The color-coding is performed in the same manner as for the accompanying online cvxNDL results. Columns of the dictionary elements in the second row are color-coded based on the genome locations of the representatives. As biologically meaningful locations can be determined only via convex methods, the top row corresponding to NMF and online NDL results is black-and-white.

https://doi.org/10.1371/journal.pcbi.1012095.g006

Classical NMF does not allow the mapping of results back to real interacting genomic regions. While the dictionary elements obtained via CMF are interpretable, they tend to mostly comprise widely spread genomic regions since they do not use the network information. The dictionary elements generated by online cvxNDL have smaller yet relevant spreads that are more likely to capture meaningful long-range interactions. In contrast to online cvxNDL, both NMF and CMF are not scalable to large datasets, rendering them unsuitable for handling current and future high-resolution datasets such as those generated by ChIA-Drop. Compared to online NDL, online cvxNDL also has a more balanced distribution of importance scores. For example, in Fig 6B , dict_0 has score 0.459, while the scores in Fig 6D are all ≤ 0.085. Moreover, akin to standard NMF, NDL fails to provide interpretable results since the dictionary elements cannot be mapped back to real interacting genomic loci.

Note that our approach is inherently an NMF-based method adapted for networks to ensure scalability, via its online nature, and interpretability, based on its convexity constraints. Besides scalability and interpretability, all the limitations of general NMF methods carry over to our method. For example, NMF approaches can be sensitive to initialization. Selection of the number of elements (or the rank of NMF) requires domain knowledge as well as heuristic search and testing. A wrong choice of the rank can lead to underfitting or overfitting the data. Furthermore, NMF does not guarantee a unique solution.

Results for other chromosomes are reported in Section D in S1 Text . Recall that both online cvxNDL and online NDL use a k -path as the template.

Reconstruction accuracy

Once a dictionary is constructed, one can use the network reconstruction algorithm from [ 15 ] to recover a subnetwork or the whole network by locally approximating subnetworks via dictionary elements. The accuracy of approximation in this case measures the “expressibility” of the dictionary with respect to the network. All methods, excluding randomly generated dictionaries used for illustrative purposes only, can accurately reconstruct the input network. For a quantitative assessment, the average precision-recall score for all methods is plotted in Table 1 . As expected, random dictionaries have the lowest scores across all chromosomes, while all other methods are of comparable quality. This means that interpretable methods, such as our online cvxNDL, do not introduce representation distortions (CMF also learns interpretable dictionaries; however, it is substantially more expensive computationally when compared to our method but does not ensure that network topology is respected). A zoomed-in sample-based reconstruction result for chr2L is shown in Fig H in S1 Text , while the reconstruction results for the entire contact maps of chr2L, chr2R, chr3L, and chr3R are available in Figs I-L in S1 Text . Additionally, for synthetic (Stochastic Block Model (SBM)) data, Fig 7 shows the reconstructed adjacency matrices for various dictionary learning methods, further confirming the validity of findings for the chromatin data. More detailed results for synthetic SBM data are available in Section C in S1 Text .

Both the x and y axes in the figures index the nodes of the synthetic network generated by the stochastic block model (SBM). The nodes are reorganized to highlight the underlying community structure. For a more quantitative analytical accuracy comparisons, see Table 1 .

https://doi.org/10.1371/journal.pcbi.1012095.g007

Methods that return interpretable dictionaries are indicated by the superscript i while methods that are scalable to large datasets are indicated by the superscript s . Online cvxNDL is both interpretable and scalable while maintaining performance on par with other noninterpretable and nonscalable methods.

https://doi.org/10.1371/journal.pcbi.1012095.t001

Gene Ontology enrichment analysis

As each dictionary element is associated with a set of representatives that correspond to real observed subnetworks, their nodes can be mapped back to actual genomic loci. This allows one to create lists of genes covered by at least one node included in the representatives.

To gain insights into the functional annotations of the genes associated with the dictionary elements, we conducted a Gene Ontology (GO) enrichment analysis using the annotation category “Biological Process” from https://urldefense.com/v3/__http://geneontology.org__;!!DZ3fjg!4VWHhuROFHcJ1bWTZ8pNxUn75T-K3BfsdTvxM1iU1hXmSGX84JcRsXyIZZS0k5Iaub9yNiansT9FS12EO52_OaGhnYs$ , with the reference list Drosophila Melanogaster . This analysis was performed for each dictionary element. Our candidate set for enriched GO terms was selected with a false discovery rate (FDR) threshold of < 0.05. Note that the background genes used for comparison are all genes from all chromosomes (the default option). We also utilized the hierarchical structure of GO terms [ 36 ], where terms are represented as nodes in a directed acyclic graph, and their relationships are described via arcs in the digraph (i.e., each “child” GO term is more specific than its “parent” term and where one child may have multiple parents).

We further refined our results by running additional processing steps. For each GO term, we identified all the paths between the term and the root node and then removed any intermediate parent GO term from the enriched GO terms set. By iteratively performing this filtering process for each dictionary element, we created a list of the most specific GO terms associated with each element. More details about the procedure are available in Section F in S1 Text .

We report the most frequently enriched GO terms for each chromosome, along with the corresponding dictionary elements exhibiting enrichment for chr3R in Fig 8 . The results for other chromosomes are available in Tables D, E, and F in S1 Text . Notably, the most frequent GO terms are related to regulatory functions, reflecting the significance of RNA Polymerase II. We also observe that dictionary elements for chr2L and chr2R are enriched in GO terms associated with reproduction and embryonic development. Similarly, chr3L and 3R are enriched in GO terms for blood circulation and responses to heat and cold.

Column ‘#’ indicates the number of dictionary elements that show enrichment for the given GO term. Also reported are up to 3 dictionary elements with the largest importance score in the dictionary, along with the “density” ρ of interactions in the dictionary element (defined in the Methods section) and median distance d med of all adjacent pairs of nodes in its representatives.

https://doi.org/10.1371/journal.pcbi.1012095.g008

We report the number of GO terms associated with each dictionary element, along with their importance scores in Tables J-M in S1 Text . Dictionary elements with higher importance scores tend to exhibit a larger number of enriched GO terms while dictionary elements with 0 enriched GO terms generally have small importance scores.

Using the entire genome as the reference is an accepted approach for GO analysis. However, it can introduce a bias due to differences in the chromosomal architectures of various chromosomes. We therefore performed an additional GO analysis where the genes within the pertinent chromosome, rather than the whole genome, are used as a reference. We implemented a Bonferroni correction and set the FDR to 0.05 (note that the results depend on the multiple-hypothesis testing correction method used). The total number of enriched GO terms across all online cvxNDL dictionaries for each of the 4 chromosomes 2L, 2R, 3L, and 3R equals 36, 19, 21, and 54, respectively.

RNA-Seq coexpression analysis

The ChIA-Drop dataset [ 17 ] used in our analysis was accompanied by a single noisy RNA-Seq replicate. To address this issue, we retrieved 20 collections of RNA-Seq data corresponding to untreated S2 cell lines of Drosophila Melanogaster from the Digital Expression Explorer (DEE2) repository. DEE2 provides uniformly processed RNA-Seq data sourced from the publicly available NCBI Sequence Read Archive (SRA) [ 23 ]. The list of sample IDs is available in Table N in S1 Text .

To ensure consistent normalization across all samples, we used the trimmed mean of M values (TMM) method [ 37 ], available through the edgeR package [ 38 ]. This is of crucial importance when jointly analyzing samples from multiple sources. We selected the most relevant genes by filtering the list of covered genes and retaining only those with more than 95% overlap with the gene promoter regions, as defined in the Ensembl genome browser. Subsequently, for each dictionary element, we collected all genes covered by it and then calculated the pairwise Pearson correlation coefficient of expressions of pairs of genes in the set. To visualize the underlying coexpression clusters within the genes, we performed hierarchical clustering, the results of which are shown in Section G in S1 Text and discussed next.

Additionally, we conducted control experiments by constructing dictionary elements through random sampling of genes from the list of all genes on each of the chromosomes. For these randomly constructed dictionaries, we carried out a coexpression analysis as described above. We observed that the mean of coexpressions of all pairs of genes in a randomly constructed dictionary element is significantly lower compared to the mean of the online cvxNDL dictionary elements. Specifically, for dictionary elements generated using online cvxNDL, the mean coexpression values for all pairs of genes covered by the 25 dictionary elements, and for each of the four chromosomes, 2L, 2R, 3L, and 3R, were found to be 0.419, 0.383, 0.411, and 0.407, respectively. The corresponding values for randomly constructed dictionaries were found to be 0.333, 0.329, 0.323, and 0.337, respectively. To determine if these differences are statistically significant, we employed the two-sample Kolmogorov-Smirnov test [ 39 ], comparing the empirical cumulative distribution functions (ECDFs) of pairwise coexpression values of the learned and randomly constructed dictionaries. The null hypothesis used was “the two sets of dictionary elements are drawn from the same underlying distribution.” The null hypotheses for all four chromosomes were rejected, with p-values equal to 3.6 × 10 −9 , 8.5 × 10 −6 , 3.6 × 10 −9 , and 2.5 × 10 −7 for chr2L, chr2R, chr3L, and chr3R, respectively (see Fig 9 ). This indicates that the learned dictionary elements indeed capture meaningful biological patterns of chromatin interactions.

The results are based on the two-sample Kolmogorov-Smirnov test, and the null hypothesis described in the main text.

https://doi.org/10.1371/journal.pcbi.1012095.g009

To further evaluate our results, we also examined the well-documented R1-R4 and T1-T4 TAD interactions on chr2L, reported in [ 17 ]. The results of the coexpression analysis for these genomic regions are reported in Fig 10 . The mean pairwise correlation between genes belonging to the R1-R4 genomic regions equals 0.422, which is comparable to the mean value 0.419 of the results obtained via online cvxNDL. We also calculated the intersection of the set of genes within the R1-R4 genomic regions and the set of genes covered by online cvxNDL dictionary elements identified for chr2L. We observed that the top 5 online cvxNDL dictionary elements cover 38 out of 85 genes in the R1-R4 genomic regions. This is to be contrasted with the results for random dictionary elements, which cover only 7 genes. Table 2 describes these and related findings in more detail.

We calculated the mean and standard deviation of absolute pairwise coexpression values, and the mean and standard deviation of coexpression values specifically for all positively correlated gene pairs. The mean coexpression values within TADs and dictionary elements are similar to each other and generally higher than those of randomly constructed dictionary elements. The x and y axis index genes that belong to the respective TAD regions or a specific dictionary element. Note that the plot (b) is of coarser resolution due to the small number of genes covered when compared to the cases (a), (c), (d).

https://doi.org/10.1371/journal.pcbi.1012095.g010

We determined the sizes of the intersections of the set of genes covered by each dictionary element and the genes in the R1-R4 genomic region and arranged them in decreasing order. The top 5 dictionary elements in this order cumulatively contain 38 out of the 85 genes within the R1-R4 genomic regions. This is in sharp contrast with randomly generated dictionary elements, where the top 5 elements with maximum intersection cover only 7 genes.

https://doi.org/10.1371/journal.pcbi.1012095.t002

We also mapped genes covered by our dictionary elements onto nodes of the STRING protein-protein interaction network [ 24 ]. These mappings allow us to determine the confidence of pairwise gene interactions. These, and related results based on FlyMine [ 40 ] data, a large gene expression repository for Drosophila Melanogaster , are available in Section G in S1 Text .

The rationale behind the STRING analysis is that gene fragments that are in physical contact are likely to be involved in the same pathway. This hypothesis, as well as the hypothesis we used for RNA-Seq based validation that genes in high-frequency contact regions are co-expressed is, still being investigated. While there is evidence to suggest that the formation of loops causes coexpression of its gene constituents, the dynamic nature of chromatin folding and the potential rewiring of chromatin contacts before transcription may make the relationship more complex [ 18 ]. This is especially the case when some promoters act as enhancers during transcription of proteins [ 41 ].

We provide further comparisons of CMF and online cvxNDL methods, the only two interpretable methods, below. To ensure a fair comparison, we select the top 10 samples with the largest convex weights that correspond to each of the CMF dictionary elements.

The total number of enriched GO terms across all online cvxNDL dictionaries for each of the 4 chromosomes 2L, 2R, 3L, and 3R is 36, 19, 21, and 54, while the numbers for CMF are 17, 7, 2, and 26, respectively. Furthermore, although fine-grained GO term comparison is not possible for the two sets (since there are many specializations of the same higher-level term), we still see that important higher-level GO terms—such as protein folding, response to stimuli, and metabolic and developmental processes—are shared by the two lists.

The mean pairwise co-expressions of genes covered by all 25 online cvxNDL and CMF dictionary elements for each of the four chromosomes analyzed (and their standard deviation) are shown in Table 3 . Also, both CMF and online cvxNDL dictionaries significantly outperform random dictionary elements. Similarly, the mean confidence values of interactions retrieved from the filtered STRING PPI network are reported in Table O in S1 Text . The RNA-Seq and PPI network analysis indicates that the interpretable dictionaries from online cvxNDL and CMF both perform similarly, while only online cvxNDL can scale to larger datasets.

https://doi.org/10.1371/journal.pcbi.1012095.t003

We also reconstructed the R1-R4 genomic regions identified in [ 17 ] using CMF dictionary elements. We observe that the top 5 dictionary elements, with the 5-highest importance scores, capture 15 of the 85 genes present in the R1-R4 genomic regions. This is to be compared to the 38 genes covered by the top 5 online cvxNDL dictionary elements and the 7 genes covered by the top 5 random dictionary elements. All 25 CMF dictionary elements together cover 45 genes, and this number is comparable to that of 54 genes covered by all 25 online cvxNDL dictionary elements. It is also significantly larger than the 11 genes covered by 25 random dictionary elements.

MCMC sampling of subnetworks (sample generation)

For NDL, it is natural to let the columns of X t be vectorized adjacency matrices of N subnetworks. Hence one needs to efficiently sample meaningful subnetworks from a (large) network. In image DL problems, samples can be generated directly from the image using adjacent rows and columns. However, such a sampling technique cannot be applied to arbitrary network data. Selecting nodes along with their one-hop neighbors at random may produce subnetworks of vastly different sizes and the results do not capture meaningful long-range interactions. It is also difficult to trim such subnetworks to uniform sizes. Furthermore, sampling a fixed number of nodes uniformly at random from sparse networks produces disconnected subnetworks with high probability and is not an acceptable approach either.

Online convex NDL (online cvxNDL)

Supporting information

S1 text. supplement pdf..

Supplemental material, including figures and tables, is available in the Supplement file. The online cvxNDL code and test datasets are available at: https://urldefense.com/v3/__https://github.com/rana95vishal/chromatin_DL/__;!!DZ3fjg!4VWHhuROFHcJ1bWTZ8pNxUn75T-K3BfsdTvxM1iU1hXmSGX84JcRsXyIZZS0k5Iaub9yNiansT9FS12EO52_XsbpA_s$ . A tool that enables readers with color-blindness to view the images using a more appropriate color palette is described at the end of the Supplement.

https://doi.org/10.1371/journal.pcbi.1012095.s001

Acknowledgments

The authors gratefully acknowledge many useful discussions with Dr. Yijun Ruan.

• View Article
• PubMed/NCBI
• Google Scholar
• 13. Peng J, Milenkovic O, Agarwal A. Online convex matrix factorization with representative regions. In: Advances in Neural Information Processing Systems; 2019. p. 13242–13252.
• 29. Agarwal S, Lim J, Zelnik-Manor L, Perona P, Kriegman D, Belongie S. Beyond pairwise clustering. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). vol. 2. IEEE; 2005. p. 838–845.