Stock Markets Analytics Zoomcamp 2025

Homework 3: Modeling for Financial Time Series Statistics

Distribution of scores and reported study time for this homework.

Submissions

44

Median total score

13

Average total score

10

Score distribution

All values are points.

Questions score

Min
1
Median
13.0
Max
13
Q1
6.8
Avg
9.8
Q3
13.0

Learning in public score

Min
-
Median
0.0
Max
3
Q1
0.0
Avg
0.4
Q3
0.0

Total score

Min
1
Median
13.0
Max
16
Q1
7.8
Avg
10.2
Q3
13.0

Time distribution

All values are hours reported by students.

Lectures

Min
1.3
Median
2.5
Max
11.0
Q1
2.0
Avg
3.7
Q3
4.8

Homework

Min
2.0
Median
5.0
Max
30.0
Q1
3.0
Avg
7.0
Q3
8.0

Question breakdown

Correctness and answer distribution per question.

1. [Dummies on Month and Week-of-Month] Top correlation of a new dummy with a boolean prediction variable?

39 / 45 correct (86.7%)

1 0.025 40 (88.9%)
2 0.035 3 (6.7%)
3 0.045 1 (2.2%)
4 0.055 0 (0.0%)

2. [Define new 'hand' rules] What's the precision score for the best new rule?

37 / 45 correct (82.2%)

1 0.610 2 (4.4%)
2 0.595 2 (4.4%)
3 0.580 37 (82.2%)
4 0.565 2 (4.4%)

3. [Unique correct predictions for pred5_clf_10] What's the total number of records of 'unique correctness' on TEST?

30 / 45 correct (66.7%)

1 2590 3 (6.7%)
2 3770 30 (66.7%)
3 5110 8 (17.8%)
4 6321 1 (2.2%)

4. [Hyperparams tuning] What's the best max_depth for a Decision Tree Classifier?

27 / 45 correct (60.0%)

1 20 6 (13.3%)
2 15 6 (13.3%)
3 10 1 (2.2%)
4 5 27 (60.0%)

5. [EXPLORATORY] What data features are missing?

44 / 45 correct (97.8%)

Answer Count
To improve predictions, I would add: – Market sentiment (Fear & Greed Index, VIX, Google Trends on the ticker) to capture psychological context. – Sector momentum (sector vs. market performance over 1M/3M) since stocks often follow their sector trends. – Intraday volatility (ATR, open/close gap) to capture price dynamics not visible in close prices. – Simple ESG or governance scores since they can influence long-term investor flows. These features would complement the existing fundamentals and ratios, helping the model capture market conditions and stock dynamics more effectively. 1
Based on the correlation analysis and Decision Tree results, here are some new indicators and approaches that could enhance the dataset for predicting stock growth: **New Indicators to Include:** 1. **Market Sentiment Indicators:** * **Reasoning:** The correlation analysis likely highlighted the impact of macroeconomic factors (like CPI, FEDFUNDS). Stock markets are heavily influenced by investor sentiment, which isn't fully captured by the current macroeconomic data. Indicators like the VIX (Volatility Index) for US markets, or similar indices for EU and India, could provide a gauge of market fear and uncertainty. News sentiment analysis (e.g., tracking news headlines related to specific sectors or the overall market) could also be a powerful predictor. * **Potential Data Sources:** Stooq.com, Yahoo Finance (though comprehensive sentiment requires more specialized APIs), news aggregators with sentiment scoring capabilities. 2. **Industry/Sector-Specific Indicators:** * **Reasoning:** The dataset includes stocks from different sectors across different regions. The performance of a tech stock might be driven by different factors than a utility stock. Including sector-specific KPIs (e.g., semiconductor sales for tech, oil prices for energy, interest rate expectations for financials) could provide more granular and relevant information. * **Potential Data Sources:** Industry reports, financial data providers like Bloomberg, Refinitiv, or even scraping specific industry websites (though this is complex). 3. **Company-Specific News and Events:** * **Reasoning:** While the dataset includes historical price data, it doesn't explicitly capture major company-specific news (like earnings reports, product launches, regulatory approvals/issues, management changes). These events can significantly impact a stock's short-term and medium-term growth. * **Potential Data Sources:** Financial news APIs (e.g., News API, Finnhub), scraping company press release sections or financial news websites. 4. **Bond Yield Spreads (beyond DGS1, DGS5, DGS10):** * **Reasoning:** The current bond yield data is limited. Spreads between different maturities (e.g., 10-year minus 2-year Treasury yield) are often seen as recession indicators and can signal shifts in investor confidence and economic outlook, which directly impacts stock valuations. Spreads between corporate and government bonds (credit spreads) also reflect risk perception. * **Potential Data Sources:** Federal Reserve Economic Data (FRED), national central bank websites for EU and India bonds, financial data providers. 5. **Currency Exchange Rates:** * **Reasoning:** Since the dataset covers stocks from the US, EU, and India, exchange rate fluctuations between USD, EUR, and INR can impact the reported earnings and perceived value of these international companies, especially for those with significant foreign operations. * **Potential Data Sources:** Federal Reserve Economic Data (FRED), European Central Bank (ECB) data, Reserve Bank of India (RBI) data, financial data APIs. **Alternative Approach:** A completely different approach could focus on a **Network-Based Analysis** of stock relationships and their diffusion patterns. * **Reasoning:** Instead of treating each stock in isolation (or primarily based on its own historical data and external macro factors), this approach views the market as a complex network where stocks are nodes and their relationships (e.g., price correlation, industry ties, supply chain links, shared investors) are edges. Predicting the growth of a stock could involve analyzing the state and trends of its directly and indirectly connected neighbors in the network. For instance, a strong upward trend in a company's key suppliers or customers might signal positive future growth for the company itself. * **Data Required for this Approach:** * **Historical Stock Price Data:** (Already have this) * **Industry Classification Data:** To identify companies in the same or related sectors. * **Supply Chain Data:** Information on key suppliers and customers for each company. This is often difficult to obtain comprehensively and accurately. * **Ownership Data:** Data on large institutional investors and which stocks they hold across the dataset. * **News/Social Media Data:** To analyze how news or sentiment about one company or sector propagates to others. * **Patent/Innovation Data:** To see if innovation in one area (e.g., a specific technology) diffuses and impacts related companies. * **Data Source for this Approach:** This approach requires combining data from multiple sources. * Historical stock prices: Yahoo Finance, Bloomberg, Refinitiv, national stock exchange APIs. * Industry classification: GICS (Global Industry Classification Standard) from financial data providers, company reports. * Supply Chain Data: Specialized databases (e.g., FactSet Revere), company annual reports, news analysis. * Ownership Data: SEC filings (for US), similar regulatory filings in EU and India, financial data providers. * News/Social Media Data: News APIs, social media platforms (requires significant processing and sentiment analysis). * Patent Data: Patent databases (e.g., Google Patents, USPTO, EPO). This network approach moves beyond traditional time-series or regression models by leveraging the interdependencies within the market. It could be particularly effective in identifying contagion effects, sector rotation, and the impact of industry-wide trends that might not be fully captured by individual stock indicators or broad macro data. The challenge lies in effectively building and representing this dynamic network and developing models (like Graph Neural Networks) that can learn from its structure and evolution. 1
Stocks from different sectors respond differently to macroeconomic and market events. Including sector/industry labels would allow for sector-based analysis and improve model accuracy. 1
Earnings Surprise (%), News/Social Sentiment 1
# Expanding the Dataset: Suggested New Indicators Based on the insights gained from correlation analysis and Decision Tree results, which often highlight the significance of macroeconomic factors, interest rates (like DGS10 and FEDFUNDS), and technical indicators, here are some additional data points that could significantly improve the predictive power of your models: ## 1. Volatility Measures (Beyond Simple Volatility) + Implied Volatility (VIX, VXN, etc.): The VIX (CBOE Volatility Index) is often called the "fear gauge" and represents the market's expectation of future volatility. Including implied volatility specific to the US (VIX), Europe (VSTOXX), and potentially India (India VIX) could provide crucial forward-looking information about market sentiment and risk. + Reasoning: High implied volatility often precedes market downturns or periods of uncertainty, while low implied volatility can indicate complacency. This offers a different dimension of risk assessment than historical volatility. + Data Source: CBOE for VIX, Eurex for VSTOXX, NSE for India VIX. ## 2. Market Breadth Indicators + Advance-Decline Line (AD Line): This technical indicator shows the difference between the number of advancing and declining stocks on a stock exchange. It can reveal underlying strength or weakness in the market that might not be apparent from price indices alone. + Reasoning: A rising AD line, even if the main index is flat, suggests broad market participation and bullish sentiment. Conversely, a falling AD line while the index rises can signal a narrowing market and potential weakness. + Data Source: Financial data providers like Bloomberg, Refinitiv, or historical data from major exchanges (NYSE, NASDAQ, Euronext, NSE). You might need to aggregate daily advance/decline data. + New Highs/Lows: The number of stocks reaching new 52-week highs versus new 52-week lows. + Reasoning: A high number of new highs and a low number of new lows indicate a strong bull market, while the opposite suggests a bear market. + Data Source: Same as AD Line, typically available from financial data providers or direct from exchanges. ## 3. Economic Sentiment and Survey Data + Consumer Confidence Index (CCI) / Business Confidence Index: These indices reflect how optimistic or pessimistic consumers and businesses are about the future state of the economy. + Reasoning: High consumer and business confidence generally lead to increased spending and investment, which can boost corporate earnings and stock prices. + Data Source: The Conference Board (US CCI), Eurostat (EU confidence indicators), RBI (India consumer confidence). + Purchasing Managers' Index (PMI): A survey-based indicator that measures the economic health of the manufacturing and services sectors. + Reasoning: PMI readings above 50 generally indicate expansion, while readings below 50 suggest contraction. This provides a timely snapshot of economic activity. + Data Source: ISM (US), Markit (EU, India, and global). ## 4. Commodity Prices (Beyond Brent Oil) + Industrial Metals (e.g., Copper): Copper is often referred to as "Dr. Copper" because its price is seen as a leading indicator of economic health due to its widespread use in various industries. + Reasoning: A rising copper price can signal increased industrial demand and economic growth, which can positively impact company earnings. + Data Source: London Metal Exchange (LME), COMEX. + Agricultural Commodities (e.g., Corn, Wheat): Prices of key agricultural commodities can impact inflation, consumer spending, and the profitability of related industries. + Reasoning: Spikes in food prices can lead to inflationary pressures and reduce discretionary consumer spending, potentially affecting broader market performance. + Data Source: Chicago Board of Trade (CBOT), Euronext. 1
Maybe we can try introducing momentum-based composite features that synthesize existing short- and medium-term growth signals.First, we can compute an Average Past Momentum feature as the mean of growth_7d, growth_14d, and growth_30d, which smooths out short-term noise and better captures consistent directional trends in price movement.Second, a Momentum Diff feature (growth_30d - growth_7d) can help detect acceleration or deceleration in momentum, signaling whether a stock's upward or downward movement is gaining or losing strength over time.Finally, a Momentum vs Benchmark feature—defined as the relative performance of a stock against its broader market to check it makes sense. 1
### Indicators to explore - **Fundamental Valuation Metrics:** Include classic ratios like **P/E**, **P/B**, **dividend yield** and **ROE** so your model can tell apart price moves driven by real earnings strength versus purely technical momentum. Trees often split first on valuation when it’s available. - **Earnings-Surprise & Analyst Revision Signals:** Add features such as **EPS surprise (%)** and **net analyst upgrades/downgrades**. Companies that consistently beat estimates or attract upward revisions tend to keep outperforming. - **News & Social-Sentiment Scores:** Aggregate daily sentiment from financial headlines (e.g. FinBERT polarity) and a simple **bull-bear ratio** from Twitter or StockTwits. Shifts in sentiment often foreshadow price inflections that technical indicators alone miss. - **Options-Market Indicators:** Bring in **implied volatility**, **30-day ATM skew**, and **put-call ratios**—options prices encode forward-looking risk appetite and hedging flows, giving you an early warning on fear/greed extremes. - **Macro-Event Flags & Regime Variables:** Add **binary flags** for earnings dates, Fed/ECB/RBI decisions and major data releases (CPI, PMI), plus rolling measures of **policy rates** and **10-year yields** across the US, EU and India to capture global regime shifts. 1
Valuation Ratios (e.g., P/E, P/S, P/B):** To gauge if a stock is over or undervalued relative to its earnings, sales, or book value 1
'growth_1d', 'growth_3d', 'growth_7d', 'growth_30d', 'growth_90d', 'growth_365d', 'growth_future_30d', 'SMA10', 'SMA20', 'volatility', 'adx', 'adxr', 'apo', 'aroon_1', 'aroon_2', 'aroonosc', 'cci', 'cmo', 'dx', 'macd', 'macdsignal', 'macdhist', 'macd_ext', 'macdsignal_ext', 'macdhist_ext', 'macd_fix', 'macdsignal_fix', 'macdhist_fix', 'mfi', 'minus_di', 'mom', 'plus_di', 'dm', 'ppo', 'roc', 'rocp', 'rocr', 'rocr100', 'rsi', 'slowk', 'slowd', 'fastk', 'fastd', 'fastk_rsi', 'fastd_rsi', 'trix', 'ultosc', 'willr', 'adosc', 'atr', 'natr', 'ht_dcperiod', 'ht_dcphase', 'ht_phasor_inphase', 'ht_phasor_quadrature', 'ht_sine_sine', 'ht_sine_leadsine', 'growth_dax_1d', 'growth_dax_3d', 'growth_dax_7d', 'growth_dax_30d', 'growth_dax_90d', 'growth_dax_365d', 'growth_snp500_1d', 'growth_snp500_3d', 'growth_snp500_7d', 'growth_snp500_30d', 'growth_snp500_90d', 'growth_snp500_365d', 'growth_dji_1d', 'growth_dji_3d', 'growth_dji_7d', 'growth_dji_30d', 'growth_dji_90d', 'growth_dji_365d', 'growth_epi_1d', 'growth_epi_3d', 'growth_epi_7d', 'growth_epi_30d', 'growth_epi_90d', 'growth_epi_365d', 'Close_y', 'growth_gold_1d', 'growth_gold_3d', 'growth_gold_7d', 'growth_gold_30d', 'growth_gold_90d', 'growth_gold_365d', 'growth_wti_oil_1d', 'growth_wti_oil_3d', 'growth_wti_oil_7d', 'growth_wti_oil_30d', 'growth_wti_oil_90d', 'growth_wti_oil_365d', 'growth_brent_oil_1d', 'growth_brent_oil_3d', 'growth_brent_oil_7d', 'growth_brent_oil_30d', 'growth_brent_oil_90d', 'growth_brent_oil_365d', 'growth_btc_usd_1d', 'growth_btc_usd_3d', 'growth_btc_usd_7d', 'growth_btc_usd_30d', 'growth_btc_usd_90d', 'growth_btc_usd_365d' 1
1. PE ratio, PB ratio, PEG ratio: Valuation multiples like forward PE or EV/EBITDA influence future returns. A stock with low PE and growing earnings often outperforms. 2. RSI, MACD, Bollinger Bands width: Many traders buy on technical signals, impacting price dynamics. 3. 1
Depth: 5, Precision: 0.6314 1
Sector-specific indicators and short interest data are missing. These could improve predictions by capturing industry trends and market sentiment. Potential sources: Yahoo Finance (sectors) and FINRA (short interest) 1
Market Sentiment and News Analytics, from sources such as Bloomberg's Sentiment Analysis (SENS), etc 1
To improve prediction accuracy, I would include earnings surprise data and analyst sentiment (e.g., EPS surprise, rating changes), as they often drive short-term stock movements. Additionally, adding global index returns (like S\&P 500 or NIFTY) and volatility measures (like VIX) can help capture broader market influence. Alternative data like Google Trends or news sentiment could further enhance signals, especially for retail-driven stocks. 1
is_positive_growth_252d_future - to predict year to year growth 1
Feature engineering to combine most influential variables such as DGS10 and DGS5, use an XGBoost model instead of Decision Trees or apply PCA to reduce dimensionality 1
We can add volume based indicators like volume accumulation/distribution patterns, add more price ratios for example, `Close_minus_Open`, `High_divided_by_Low` 1
one would be the optimum amount of time to monitor growth 1
To improve predictions, I would add: – Other Macroeconomic indicators (interest rates, inflation, unemployment) to provide regional economic context. – Sector classification to group stocks by industry, allowing the model to capture shared trends within sectors. – News or social media sentiment (e.g., FinBERT scores, Reddit/Twitter trends) to reflect investor expectations and behavioral shifts. – Earnings surprise indicators to highlight key events that often trigger sharp price movements. – Currency exchange rates (e.g., INR/USD, EUR/USD) to normalize returns across different markets and capture currency risk. These features would complement the existing financial indicators and improve the model’s ability to understand both global economic conditions and stock-specific behavior. 1
Instead of adding more variables, we could: Replace the single decision-tree with a gradient-boosted tree ensemble (e.g. XGBoost or LightGBM). Boosting handles non-linear interactions automatically and usually beats a single tree by 5–15 ppt precision without manual feature engineering. Use time-series cross-validation (rolling-window) to avoid look-ahead bias and tune hyper-parameters with Bayesian optimisation (e.g. Optuna). Explain the ensemble with SHAP values to keep interpretability at a similar level to the current tree visualisation. 1
DGS10; DGS5; DGS1; FEDFUNDS; growth 90 and 365 days are most important; we can also add Exchange Rates 1
Maybe adding SMA and Volatility to smooth noise and quantify variability, often showing clear correlation. 1
Fundamental data - such as earnings, book value, profits (and all the associated metrics such as P/E etc.)) 1
##### **Answer**: Based on the correlation analysis and decision tree results, I can see several patterns that suggest specific types of missing data that could improve the model's predictive power. Here are some suggestions: ## Missing Macroeconomic Indicators **1. Regional Interest Rates and Monetary Policy** Since DGS1, DGS5, DGS10, and FEDFUNDS show strong correlations, the dataset would benefit from: - **European Central Bank (ECB) rates** - Main refinancing rate, deposit facility rate - **Reserve Bank of India (RBI) rates** - Repo rate, reverse repo rate - **10-year government bond yields** for Germany (Bund) and India - **Yield curve spreads** (10Y-2Y, 10Y-3M) for each region **2. Inflation and Economic Growth Metrics by Region** Currently only US GDP and CPI data is included: - **Eurozone HICP (Harmonized Index of Consumer Prices)** - **India CPI and WPI (Wholesale Price Index)** - **Regional GDP growth rates** (Eurozone, India) - **PMI (Purchasing Managers' Index)** for manufacturing and services in all three regions ## Missing Market Structure Indicators **3. Volatility and Risk Measures** Given the strong correlation with oil and major indices: - **VIX (S&P 500 volatility index)** - **VSTOXX (Euro Stoxx 50 volatility index)** - **India VIX** - **Currency volatility** (EUR/USD, USD/INR, EUR/INR) **4. Cross-Asset Correlations** - **USD Index (DXY)** - affects all international investments - **Regional currency performance** against USD - **Sector rotation indicators** - relative performance of defensive vs. cyclical sectors ## Missing Fundamental Data **5. Earnings and Valuation Metrics** - **Forward P/E ratios** by region/sector - **Earnings revision trends** (upgrades vs. downgrades) - **Dividend yield spreads** relative to government bonds - **Book value growth rates** ## Alternative Data Sources **6. Sentiment and Flow Data** - **Google Trends** for economic terms ("recession", "inflation", "stock market") - **ETF flows** into regional equity funds - **Insider trading activity** (aggregate buying/selling by insiders) - **Analyst sentiment scores** from financial news ## Data Sources - **FRED (Federal Reserve Economic Data)** - for additional US and some international macro data - **ECB Statistical Data Warehouse** - for Eurozone indicators - **Reserve Bank of India database** - for Indian economic data - **Yahoo Finance/Alpha Vantage** - for additional market indicators - **CBOE** - for volatility indices - **Bloomberg/Refinitiv** - for sentiment and flow data (if accessible) ## Reasoning The current model shows that **long-term trends** (365-day growth rates) and **interest rate environment** are most predictive. This suggests that: 1. **Macro regime changes** are crucial - adding regional monetary policy data would capture these better 2. **Cross-regional spillovers** matter - the correlation with oil and major indices suggests global interconnectedness 3. **Risk appetite cycles** drive returns - volatility indices and sentiment measures would capture this 4. **Seasonal patterns** (October/November effect) might be enhanced with earnings season timing and economic calendar events The missing regional specificity in macro data is particularly important since the dataset covers three distinct economic regions with different monetary policies, growth cycles, and market structures. 1
The current dataset highlights long-term price trends and interest rate sensitivity as key drivers. However, it lacks critical dimensions that reflect regional macro conditions, market structure, and investor behavior—limiting its predictive depth across the US, EU, and Indian markets. Key areas for enhancement include: - Regional macroeconomic indicators: Incorporate central bank balance sheets to capture policy beyond rate levels; add labor and credit data such as unemployment rates, wage growth, loan growth, and non-performing loan (NPL) ratios; and use inflation proxies like real estate and commodity indices (e.g., gold, copper). - Market structure and liquidity signals: Include bid-ask spreads and turnover ratios to assess market stress and liquidity; use advance-decline ratios to measure market breadth; and consider cross-asset signals such as USD strength, commodity price trends, and housing market data. - Investor sentiment and behavioral indicators: Track retail trading activity through volume changes in retail-driven stocks; extract sentiment from regional news using NLP techniques; and assess options positioning via put-call ratios and changes in open interest. 1
https://github.com/HaChan/stock-markets-analytics-zoomcamp-2025/blob/master/hw3.ipynb 1

Calculated: 15 July 2025, 20:34