# LLMS.txt for jgtpy
# Market Data Services, Indicator Data Processing, and Automated Data Refresh Service

GET /README.md
GET /docs/jgtservice_implementation.md
GET /docs/README.md
GET /docs/CDSSvc_purpose.md
GET /docs/CDS_purpose.md
GET /docs/IDS_purpose.md
GET /docs/CDS_data_columns.md
GET /docs/IDS_data_columns.md

## Package Purpose
jgtpy serves as the core data services layer for the JGT platform, providing market data processing, technical indicator calculation, structured data output for signal detection systems, and automated data refresh services with cloud distribution.

## Core Data Services

### Signals

* Bellow is a CSV of the various signals that this package aim to serve with their corresponding columns in CDS data:

```
Name;Title;Description;Question;Notes
fdb;Fractal Divergent Bar Code;Fractal Divergent Bar Code (contains the signal value either buy, sell or nothing);;" bdb"
fdbs;Fractal Divergent Bar Sell;Fractal Divergent Bar Sell;;" Bearish Divergent Bar"
fdbb;Fractal Divergent Bar Buy;Fractal Divergent Bar Buy;;" Bullish Divergent Bar"
acs;AC Deceleration Sell;AC Deceleration Sell;;" "
acb;AC Acceleration Buy;AC Acceleration Buy;;" "
fs;Fractal Sell;Fractal Sell;Can the representation be optimized ? Enhanced ?;" "
fb;Fractal Buy;Fractal Buy;;" "
zlcb;Zero Line Crossing Buy;Zero Line Crossing Buy;;" "
zlcs;Zero Line Crossing Sell;Zero Line Crossing Sell;;" "
zcol;Zero Line Crossing;Zero Line Crossing (Not sure if this is a signal);How many bars were there since last cross signal of the opposite direction ? How is the relationship of this signal profit and this number of bars ?;" Number of bars before last cross when another type of signal was generated could be a learning"
sz;Zone Signal Sell;Zone Signal Sell;;" "
bz;Zone Signal Buy;Zone Signal Buy;;" "
ss;Saucer Sell;Saucer Sell;;" "
sb;Saucer Buy;Saucer Buy;;
```
(it might be somewhere later but for now that will tell LLM about them easiely)


### PDS (Price Data Service)
- **Purpose**: Raw market data storage and retrieval
- **Source**: Market data from jgtfxcon connectivity layer
- **Format**: OHLCV (Open, High, Low, Close, Volume) with Date index
- **Storage**: `$JGTPY_DATA/pds/` and `$JGTPY_DATA_FULL/pds/`

### IDS (Indicator Data Service)
- **Purpose**: Technical indicator calculation and standardization
- **Key Module**: `JGTIDS.py` and `jgtapyhelper.py`
- **Indicators**: Bill Williams suite (Alligator, AO/AC, Fractals, MFI)
- **Output**: Standardized indicator columns for signal analysis

### CDS (Chart Data Service)
- **Purpose**: Visual chart data preparation and analysis
- **Integration**: Combines PDS + IDS with additional signal columns
- **Usage**: Chart generation and visual analysis

### ADS (Analysis Data Service)
- **Purpose**: Advanced analytical data preparation
- **Features**: Multi-timeframe analysis and pattern detection

## Key Components

### JGTIDS.py - Indicator Data Service
Primary module for indicator calculation:

```python
from jgtpy.JGTIDS import ids_add_indicators, tocds

# Generate IDS with indicators
df_with_indicators = ids_add_indicators(price_data, quiet=True)

# Generate full CDS with signals
df_cds = tocds(price_data, quiet=True)
```

**Core Functions**:
- `ids_add_indicators()` - Add technical indicators to DataFrame
- `tocds()` - Complete conversion to Chart Data Service format
- `_ids_add_fdb_column_logics_v2()` - FDB signal detection logic

### jgtapyhelper.py - Service Layer
Helper functions and service wrappers:

```python
from jgtpy import jgtapyhelper as th

# Create IDS from PDS file
df_ids = th.create_from_pds_file(rq=ids_request)

# Read existing IDS data
df_ids = th.read_ids(instrument, timeframe)

# Get current bar data
current_bar = th.read_ids_currentbar(instrument, timeframe, use_full=False)
```

**Key Functions**:
- `create_from_pds_file()` - Generate IDS from PDS data
- `read_ids()` - Load existing IDS data
- `get_bar_at_index()` - Extract specific bars for analysis

### JGTIDSRequest Configuration
Controls indicator calculation and data processing:

```python
from jgtpy.JGTIDSRequest import JGTIDSRequest

rq = JGTIDSRequest()
rq.instrument = "EUR/USD"
rq.timeframe = "H4"
rq.mfi_flag = True              # Enable MFI calculation
rq.balligator_flag = True       # Enable Big Alligator
rq.talligator_flag = True       # Enable Tide Alligator
rq.keep_bid_ask = False         # Remove bid/ask columns
```

## CLI Interface

### Primary Commands
```bash
# Generate IDS data
jgtpycli ids -i EUR/USD -t H4 --fresh

# Generate CDS with charts
jgtpycli cds -i SPX500 -t D1 --full

# Generate ADS analysis
jgtpycli ads -i GBP/USD -t H1 --fresh
```

### Integration with Platform
```bash
# Called by jgtml for fresh data
jgtapp ids EUR/USD H4

# Called by scanning systems
jgtapp cds SPX500 D1 --fresh
```

## Data Processing Pipeline

### Standard Workflow
```
Raw Price Data (PDS) → Indicator Calculation (IDS) →
Signal Processing (CDS) → Analysis Output (ADS) →
Chart Generation → Signal Detection
```

### Used by jgtml Signal Detection
```python
# jgtml calls jgtpy for data preparation
from jgtpy import jgtapyhelper as th

# Get fresh IDS data for signal scanning
df = th.create_from_pds_file(instrument, timeframe, use_fresh=True)

# Extract signal bars for analysis
signal_bar, current_bar = get_last_two_bars(df)
```

## Column Standards

### IDS Column Naming (from jgtutils.jgtconstants)
```python
# Alligator indicators
JAW, TEETH, LIPS                # Regular Alligator (13-8-5)
BJAW, BTEETH, BLIPS             # Big Alligator (89-55-34)
TJAW, TTEETH, TLIPS             # Tide Alligator (377-233-144)

# Oscillators
AO, AC                          # Awesome/Accelerator Oscillators

# Fractals (multiple degrees)
FH, FL, FH3, FL3, FH5, FL5      # Fractal highs/lows
FH8, FL8, FH13, FL13            # Higher degree fractals
# ... up to FH89, FL89

# Market Facilitation Index
MFI                             # Bill Williams MFI

# Signal columns
FDB, FDBB, FDBS                 # Fractal Divergent Bar signals
ZONE_SIGNAL                     # Market zone identification
```

### Data Types and Formatting
- **Numerical precision**: Configurable rounding (default 10 decimal places)
- **Boolean conversion**: Automatically convert bool columns to int
- **Date handling**: Pandas datetime index with timezone support
- **Missing data**: Configurable NaN handling and dropna options

## Integration with Other Packages

### Data Flow from jgtfxcon
```python
# jgtfxcon provides raw market data
import jgtfxcon
raw_data = jgtfxcon.getPH(instrument, timeframe, quote_count)

# jgtpy processes into indicators
from jgtpy import jgtapyhelper as th
indicators_data = th.create_from_df(raw_data)
```

### Data Flow to jgtml
```python
# jgtml uses jgtpy-processed data for signals
from jgtpy.JGTIDS import _ids_add_fdb_column_logics_v2

# Get processed data with FDB signals
df_with_fdb = _ids_add_fdb_column_logics_v2(indicators_data)
```

### Usage in jgtapp Workflows
```python
# jgtapp orchestrates data processing workflows
from jgtpy.jgtpyconstants import IDSCLI_PROG_NAME, CDSCLI_PROG_NAME

# Generate fresh IDS data
subprocess.run([IDSCLI_PROG_NAME, '-i', instrument, '-t', timeframe, '--fresh'])

# Generate CDS for charting
subprocess.run([CDSCLI_PROG_NAME, '-i', instrument, '-t', timeframe, '--full'])
```

## Configuration and Settings

### Environment Variables
- `JGTPY_DATA` - Standard data storage path
- `JGTPY_DATA_FULL` - Full historical data storage path

### Configuration Files
- Settings loaded via jgtutils.jgtcommon configuration system
- Column removal specifications for data cleanup
- Indicator parameter defaults and overrides

## Performance and Optimization

### Caching Strategy
- **File-based caching** for processed indicator data
- **Timestamp validation** for cache freshness
- **Configurable cache paths** for different data types

### Data Processing Optimization
- **Vectorized calculations** using pandas and numpy
- **Conditional indicator calculation** based on available bars
- **Memory-efficient processing** for large datasets

## Development State

### Production Components
- **IDS Generation**: Stable and production-ready
- **Data Services**: Core functionality complete
- **CLI Interface**: Full feature set available

### Active Development
- **Multi-timeframe optimization** for faster processing
- **Enhanced caching strategies** for improved performance
- **Integration improvements** with jgtml signal detection

### Integration Points
- **jgtfxcon**: Receives raw market data
- **jgtml**: Provides processed data for signal detection
- **jgtutils**: Uses configuration and utility functions
- **jgtapy**: Leverages indicator calculation engine

# Examples

The `examples/` directory contains runnable scripts demonstrating each command
line interface. Each subfolder has a `run.sh` script and README describing the
before/after state when executing that CLI tool.

---

*jgtpy serves as the essential data processing backbone of the JGT platform, transforming raw market data into structured, indicator-rich datasets ready for advanced signal analysis and trading decisions.*


## JGT Data Refresh Service (v0.6.0)

The JGT Data Refresh Service (`jgtservice`) provides automated data processing with parallel execution, intelligent scheduling, and cloud distribution.

### Service Components
- **JGTServiceManager**: Main service orchestration and lifecycle management
- **JGTScheduler**: Timeframe-based task scheduling using jgtutils patterns
- **ParallelProcessor**: Multi-threaded data processing with configurable workers
- **CloudUploader**: Modernized Dropbox integration for automatic uploads
- **JGTServiceAPI**: FastAPI web service with RESTful endpoints

### CLI Usage
```bash
# One-time refresh
jgtservice --refresh-once -i EUR/USD -t H1

# Continuous daemon mode
jgtservice --daemon --all

# Web API server
jgtservice --web --port 8080

# Service status
jgtservice --status
```

### Web API Endpoints
```
GET /api/v1/data/{instrument}/{timeframe}     # Retrieve CDS data
GET /api/v1/data/{instrument}/{timeframe}/latest  # Latest data point
GET /api/v1/status                           # Service status
GET /api/v1/health                           # Health check
POST /api/v1/refresh                         # Trigger refresh
GET /api/v1/metrics                          # Processing metrics
GET /docs                                    # API documentation
```

### Installation
```bash
# Base functionality
pip install -e .

# With web service dependencies
pip install -e ".[serve]"
```

### Configuration
Uses environment variables for configuration:
- `JGTPY_DATA` / `JGTPY_DATA_FULL` - Data storage paths
- `TRADABLE_TIMEFRAMES` / `HIGH_TIMEFRAMES` / `LOW_TIMEFRAMES` - Timeframe configuration
- `JGTPY_DROPBOX_APP_TOKEN` - Upload authentication
- `JGTPY_SERVICE_*` - Service-specific settings

### Performance
- 50%+ faster than sequential processing
- EUR/USD H1: ~8 seconds processing time
- Memory stable during extended runs
- Individual failure isolation

For complete documentation: `docs/jgtservice_implementation.md`

------
# command line interface (CLI) for jgtpy guidance

* guidecli_jgtpy --list  # Shows all 10 sections including new ones
* guidecli_jgtpy --section jgtmouthwater  # Shows new comprehensive guide