1.2.10. Transfer Service 1.0 Implementation

The Transfer service provides a unified interface for file exchange between remote locations and the local device. It serves multiple purposes:

  • Downloading application updates

  • Exchanging monitoring and administrative data

  • Uploading device information and logs

  • Managing content distribution for modules

  • Standardizing HTTP communication (POST, GET, etc.) for other services

1.2.10.1. Store Concept

The service uses a flexible “store” abstraction to manage different storage locations. This design allows:

  • Configuration-based location management

  • Multiple storage backends

  • Context-specific storage (logs, updates, monitoring)

  • Easy switching between storage implementations

1.2.10.1.1. Store Configuration

Stores are configured using JSON serialized configuration. Example from a production setup:

{
    "store_name": "config",
    "bucket_name": "etabelone-dev-etabelonedevbootstrape71b79bd-kfzemjujlgbe",
    "region": "eu-west-1",
    "prefix": ""
}

1.2.10.2. Service Configuration

The service can be configured through environment variables and service properties:

1.2.10.2.1. Environment Variables

Environment Variable

Description

Example

AWS_ACCESS_KEY_ID

AWS Access key for authentication

AKIATCKAQOQG77TZDSOH

AWS_SECRET_ACCESS_KEY

AWS Secret key for authentication

teEFXxZFWy10AzvQGSLTwGzqPq67hnY3rYk3MHzvV

S3_STORE_CUSTOM_HOSTANDPORT_ENDPOINT

if aws is not used, this define the host and port to call

S3_STORE_CUSTOM_HTTP_SCHEME

if S3_CUSTOM STORE is used, this define the protocol (can be either http or https)

STORE_CONFIGURATION

JSON serialized store configuration

See store configuration example above

PROXY_DEFINITION

Proxy server configuration for network requests

See proxy configuration examples below

1.2.10.2.2. Proxy Configuration

The transfer service supports proxy configuration through the PROXY_DEFINITION environment variable. This allows the service to route network requests through proxy servers for enhanced security, access control, or network routing.

1.2.10.2.2.1. Supported Proxy Types

The service automatically detects the proxy type from the URL scheme:

  • HTTP Proxy: http://proxy.example.com:8080

  • HTTPS Proxy: https://proxy.example.com:8443

  • SOCKS4 Proxy: socks4://proxy.example.com:1080

  • SOCKS5 Proxy: socks5://proxy.example.com:1080

1.2.10.2.2.2. Basic Proxy Configuration

# HTTP proxy without authentication
export PROXY_DEFINITION="http://proxy.example.com:8080"

# HTTPS proxy without authentication
export PROXY_DEFINITION="https://proxy.example.com:8443"

# SOCKS5 proxy without authentication
export PROXY_DEFINITION="socks5://proxy.example.com:1080"

1.2.10.2.2.3. Proxy with Authentication

# HTTP proxy with username/password authentication
export PROXY_DEFINITION="http://username:password@proxy.example.com:8080"

# SOCKS5 proxy with authentication
export PROXY_DEFINITION="socks5://user123:pass456@proxy.example.com:1080"

# HTTPS proxy with authentication
export PROXY_DEFINITION="https://admin:secure_pass@proxy.example.com:8443"

1.2.10.2.2.4. Environment Variable Substitution

For enhanced security, you can use environment variables for sensitive credentials:

# Set proxy credentials in separate environment variables
export PROXY_USER="username"
export PROXY_PASS="password"

# Use environment variables in proxy definition
export PROXY_DEFINITION="http://${PROXY_USER}:${PROXY_PASS}@proxy.example.com:8080"

1.2.10.2.2.5. Proxy Configuration Examples

1.2.10.2.2.5.1. Example 1: Corporate HTTP Proxy
export PROXY_DEFINITION="http://corp-proxy.company.com:3128"
1.2.10.2.2.5.2. Example 2: SOCKS5 Proxy with Authentication
export PROXY_DEFINITION="socks5://user:pass@socks.company.com:1080"
1.2.10.2.2.5.3. Example 3: HTTPS Proxy for Secure Connections
export PROXY_DEFINITION="https://secure-proxy.company.com:8443"
1.2.10.2.2.5.4. Example 4: Local Development Proxy
export PROXY_DEFINITION="http://localhost:8080"

1.2.10.2.2.6. Proxy Behavior

  • Automatic Detection: Proxy type is automatically determined from the URL scheme

  • Fallback: Unknown schemes default to HTTP proxy behavior

  • Authentication: Supports both URL-embedded and environment variable credentials

  • Protocol Support: Works with all transfer service operations (HTTP requests, S3 transfers, etc.)

1.2.10.2.2.7. Security Considerations

  • Credential Storage: Avoid hardcoding passwords in scripts or configuration files

  • Environment Variables: Use environment variables for sensitive proxy credentials

  • Network Security: Consider using HTTPS proxies for sensitive data transmission

  • Access Control: Ensure proxy servers are properly configured and secured

1.2.10.2.2.8. Troubleshooting

If proxy configuration issues occur:

  1. Verify Proxy Server: Ensure the proxy server is accessible and running

  2. Check Credentials: Verify username/password if authentication is required

  3. Network Access: Confirm network access to the proxy server

  4. Protocol Support: Ensure the proxy server supports the required protocol

  5. Port Configuration: Verify the proxy server port is correct and accessible

1.2.10.2.3. Service Properties

The service can be configured as part of the application definition:

{
    "application_id": "aws_factory_transfert-1.0",
    "service_name": "transfer",
    "service_port": 8015,
    "application_description": "transfer files to aws, with default aws device enrollment access",
    "additional_properties": {
        "AWS_ACCESS_KEY_ID": "AKIATCKAQOQG77TZDSOH",
        "AWS_SECRET_ACCESS_KEY": "teEFXxZFWy10AzvQGSLTwGzqPq67hnY3rYk3MHzvV",
        "AWS_STORE_CONFIGURATION": "[{\"store_name\":\"config\",\"bucket_name\":\"etabelone-dev-etabelonedevbootstrape71b79bd-kfzemjujlgbe\",\"region\":\"eu-west-1\",\"prefix\":\"\"}]",
        "PROXY_DEFINITION": "http://proxy.example.com:8080"
    }
}

**Note**: The `PROXY_DEFINITION` can also be configured as an environment variable for easier deployment and security management.

1.2.10.3. Store Implementations

The service currently supports two storage backends:

  1. Filesystem Store

    • Local file/folder storage

    • Used for testing and validation

    • Simple configuration with local paths

  2. AWS S3 Store

    • Cloud storage implementation

    • Production-ready configuration

    • Supports multiple buckets and regions

    • Requires AWS credentials and bucket configuration

1.2.10.4. Methods

1.2.10.4.1. HTTP Request Method

The service provides a standardized way to make HTTP requests to external servers:

call_service('transfer', 'http_request', 'https://www.google.fr', 'GET', {}, "")

Response format:

{
    "headers": [...],
    "httpcode": 200,
    "response_body": "..."
}

1.2.10.5. Usage Examples

1.2.10.5.1. AWS S3 Store Configuration

For AWS S3 storage, configure the store with:

  • Store name (e.g., “config”)

  • Bucket name (e.g., “etabelone-dev-…”)

  • AWS region (e.g., “eu-west-1”)

  • Optional prefix for object organization

1.2.10.5.2. HTTP Communication

The service standardizes HTTP communication for other modules:

  • GET requests for data retrieval

  • POST requests for data submission

  • Support for custom headers and body content

  • Response handling with status codes and headers

  • Proxy Support: All HTTP requests automatically use the configured proxy if PROXY_DEFINITION is set

1.2.10.5.3. Example of http_request calling from an other service/module

sending and http GET request to an external webserver,

example :

call_service('transfer','http_request','https://www.google.fr','GET',{},"")

this returns :

{
  "headers": [
    "date: Sun, 23 Feb 2025 15:57:58 GMT",
    "expires: -1",
    "cache-control: private, max-age=0",
    "content-type: text/html; charset=ISO-8859-1",
    "content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-7vwSUhE3dcP7hy_mah_bDg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp",
    "accept-ch: Sec-CH-Prefers-Color-Scheme",
    "p3p: CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"",
    "server: gws",
    "x-xss-protection: 0",
    "x-frame-options: SAMEORIGIN",
    "set-cookie: AEC=AVcja2ek4TuDGhyVR32F4wio5orc3vTLcsRvNye-GRM3i_8YyeP2EIvNr8Q; expires=Fri, 22-Aug-2025 15:57:58 GMT; path=/; domain=.google.fr; Secure; HttpOnly; SameSite=lax",
    "set-cookie: __Secure-ENID=25.SE=iwktx4Am0W8OAF3ucW4LzTDvBpXYMPXa162ZlQP1HBDPG6RhYMGfLv7ERi04bHspPt6qG-Q8Y6fTHQMIt83iDi2_9Z3jlNDxCi1MSap9HlaxCTKX7JERTO81u8xr1F_JRyZyoYzOu5RSSYyPlgJJHQ_jj_-qxXWklb3ndC8Iwc9PYrG-loTm6NrXZSMA0e6DP4N3oRBzY4VfjSJbRS3YNzwfZL2IZigHVx53aIldSCX1QUsmi1P0; expires=Thu, 26-Mar-2026 08:16:16 GMT; path=/; domain=.google.fr; Secure; HttpOnly; SameSite=lax",
    "alt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000",
    "accept-ranges: none",
    "vary: Accept-Encoding",
    "transfer-encoding: chunked"
  ],
  "httpcode": 200,
  "response_body": "<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"fr\"><head><meta content=\"text/html; charset=UTF-8\" http-equiv=\"Content-Type\"><meta content=\"/images/branding/googleg/1x/googleg_standard_color_128dp.png\" itemprop=\"image\"><title>Google</title><script nonce=\"7vwSUhE3dcP7hy_mah_bDg\">(function(){var _g={kEI:'hkW7Z9fXN62fkdUPn8PrAQ',kEXPI:'0,56842,145950,3497542,615,435,538661,2872,2891,43028,30022,6397,9708,78219,266577,247319,11814,30911,5230280,11407,15,47,1,8834897,27977677,25228681,46375,65746,10983,15164,8182,5936,65167,6749,9179,14700,7034,2105,4599,328,6226,1117,9301,23893,29854,1341,11792,1915,8214,7421,12132,9170,33,26708,10671,14560,6779,2987,1501,3853,41,2956,960,9723,1,6741,4106,4451,3,3282,11086,479,1808,3993,951,2149,4614,5774,4310,2371,338,925,2,738,6506,2,1459,10960,825,3894,11805,3261,458,1640,4347,714,1789,1254,55,1027,1,212,616,221,1,4307,5,3282,569,918,366,1083,423,1,2695,474,186,186,2070,386,538,2,1648,2,206,354,305,351,527,3,2699,524,1365,8,109,41.......................... r(\"click\",G)});}).call(this);</script></body></html>"
}