2013 ModelSelectioninMarkovianProces

• (Hallak et al., 2013) ⇒ Assaf Hallak, Dotan Di-Castro, and Shie Mannor. (2013). “Model Selection in Markovian Processes.” In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ISBN:978-1-4503-2174-7 doi:10.1145/2487575.2487613

Subject Headings:

Cited By

• http://scholar.google.com/scholar?q=%222013%22+Model+Selection+in+Markovian+Processes
• http://dl.acm.org/citation.cfm?id=2487575.2487613&preflayout=flat#citedby

Quotes

Author Keywords

• Dynamic mailing policies; marketing; markov decision processes; markov processes; model selection; reinforcement learning

Abstract

When analyzing data that originated from a dynamical system, a common practice is to encompass the problem in the well known frameworks of Markov Decision Processes (MDPs) and Reinforcement Learning (RL). The state space in these solutions is usually chosen in some heuristic fashion and the formed MDP can then be used to simulate and predict data, as well as indicate the best possible action in each state. The model chosen to characterize the data affects the complexity and accuracy of any further action we may wish to apply, yet few methods that rely on the dynamic structure to select such a model were suggested.

In this work we address the problem of how to use time series data to choose from a finite set of candidate discrete state spaces, where these spaces are constructed by a domain expert. We formalize the notion of model selection consistency in the proposed setup. We then discuss the difference between our proposed framework and the classical Maximum Likelihood (ML) framework, and give an example where ML fails. Afterwards, we suggest alternative selection criteria and show them to be weakly consistent. We then define weak consistency for a model construction algorithm and show a simple algorithm that is weakly consistent. Finally, we test the performance of the suggested criteria and algorithm on both simulated and real world data.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2013 ModelSelectioninMarkovianProces	Assaf Hallak Dotan Di-Castro Shie Mannor			Model Selection in Markovian Processes				10.1145/2487575.2487613		2013