Technology
Technical approach
Sentinel is built around a simple idea: reliable research depends on reliable data and on results that can be reproduced.
Data pipeline
Market data is collected from public sources, then validated, reconciled, and transformed into consistent datasets. Each stage records what was done and to which input, so a dataset can be traced back to its origin. Collection runs as scheduled work, and outputs are stored in a form suited to long-term historical use rather than short-lived queries.
Historical data handling
Long histories are large and keep growing. Sentinel is designed around storing historical data durably and cost-consciously, and processing it on demand when research requires it. This keeps idle cost low and avoids running infrastructure that is not actively doing work.
Reproducibility
Research is organized so that a given dataset version and a given method produce a recorded result. Experiments reference the exact data they used, and when data is updated or corrected, earlier results stay tied to the version they were produced from. This makes it possible to distinguish a change in the data from a change in the method.
Principles
Engineering principles
- Data provenance is recorded, not assumed.
- Results are reproducible, or they are not treated as results.
- Compute is used on demand; idle infrastructure is avoided.
- Data quality problems are treated as findings, not inconveniences.
- Simplicity in infrastructure is preferred over unnecessary components.
Scope of this page
This page describes the project's general approach only. Specific internal architecture, research methods, infrastructure configuration, and datasets are not published.