Matomo

Cryptographic IP protection for AI/ML product | Cossack Labs
Case studies
Cryptographic IP protection for AI/ML product

Cryptographic IP protection for AI/ML product

DEC 2020 - JUNE 2021 Mobile application AI / ML
Cryptographic IP protection for AI/ML product

Overview

Brought to the limelight, the product team behind the [REDACTED]'s application was looking for a high-profile and heavy cover for their technology. This product uses AI/ML technology that alters media data – photos and videos. The technology is based on artificial neural networks, which are heavily optimised and improved.

The [REDACTED] was striving to safeguard their unique IP and asked Cossack Labs to improve AI/ML security and build ML protection technology. They got it by applying the defence in depth approach to protect their IP and sensitive parts of dataflow.

Industry

  • AI/ML
  • Media

Technology stack

  • iOS native, Android apps
  • GCP
  • Python, Go backend
  • ML / TensorFlow

Regulations

  • CCPA, GDPR, local privacy regulations
  • Encryption Export Regulations

Challenges

[01]

Protecting unique IP (ML models) against leakage and misuse.

Soon after the launch, this highly sophisticated and powerful machine learning technology enjoyed tremendous viral growth and popularity. The team faced the need to optimise the system design under load, meanwhile, their app became an object of envy for attackers and plagiarism.

[02]

Overnight success turned into a challenge:

how to secure this state-of-the-art tech without affecting the team and app performance, and stay adamant in meeting data security requirements for IP and PII. The team needed help in building specialized security defences to protect their ML models, APIs, and security coverage of the sensitive data life cycle across their apps, services, databases, and data lakes.

[03]

After careful study of companies

that design security systems and work with cryptography, they asked Cossack Labs' engineers for security advisory and engineering.

Technology requirements

Technology requirements

IP protection system

The protection system for TensorFlow ML models should minimize their lifetime and make them difficult to misuse. This includes on-device protection and API anti-fraud system.

Security that doesn't ruin UI/UX

Security measures should be seamlessly integrated across mobile apps, API, and backend infrastructure. End users shouldn't feel the struggle.

Flexible cryptographic layer

Cryptographic layer should work across platforms and be easy to maintain, giving the Customer's team the necessary flexibility for improving their product.

Our approach

Security as business value

Shotgun judgments and immediate decisions only do harm when trying to solve novel sophisticated problems. To ensure that we're focusing on issues of real relevance and priority to the Customer's business model, we started from risk assessment and threat modelling.

Risk analysis

At this stage, the Customer's team got equipped with a risk analysis of their applications and infrastructure specific needs, as well as a security strategy, all allowing them to prioritize security measures.

Moving hand in hand with dev team

Then, together with the app team, we've focused on incorporating security into all steps of SSDLC: designing a well-rounded set of security controls and processes that enable IP protection, PII protection, and application security.
Our approach

Solution

We've designed and built defence in depth security measures focused on ML models protection against IP leakage and reverse engineering techniques.

Elegant cryptographic scheme to link ML models with exact users:

  • We built a cryptographic system with multi-layered encryption and a tailored key scheme that encrypts ML model per exact user.
  • The cryptographic layer uses a combination of symmetric and asymmetric primitives (HPKE-like scheme), as well as a number of supporting crypto-schemes for various parts of the ML flow.
  • To decrease server-side load and prevent building a complicated PKI architecture, we designed a key management scheme using ephemeral keys (no need to protect key storage if you don't store keys).
  • ML models are encrypted on a server-side using unique keys per usage. ML models are stored encrypted until they get into the mobile app. Mobile app decrypts ML model and re-encrypts it for storage using the hardware-backed cryptographic module.
  • Each ML model is encrypted as file and has layers of encrypted weights inside, making it useless to intercept.
  • The cryptographic system is based on a free open-source cryptographic library Themis that provides a single API across programming languages while hiding cryptographic details under the hood.
  • The resulting scheme encrypts ML model with DRM-like access control with encryption keys linked to the exact user on the exact device.

Reverse engineering protections on mobile devices:

  • Built-in protections that prevent running the app on jailbroken/rooted device, or in debug mode. The reverse engineering protections are based on the latest iOS/Android behaviour and balance between paranoia and false positives.
  • Except for using Keychain/Keystore, the hardware-backend cryptographic modules are used: Secure Enclave and hardware-backed Keystore. They provide more security guarantees for stored encryption keys used for model re-encryption.
  • Mobile application utilizes device attestation API: SafetyNet for Android and App Attest for iOS to prevent installing the application from untrusted sources.

API protection and anti fraud system:

  • Since nothing exists in a vacuum, we made sure that new IP protection controls rely on a solid foundation.
  • We improved API security and user authentication, ensured that every API call is authenticated, introduced API limits, throttling, firewalling and monitoring.
  • To minimize chances that ML models could be leaked from cloud storage, their TTL is set to minutes.
  • We suggested and co-designed the anti-fraud system, which addresses both security concerns and prevents resource spending on malicious users.
  • The anti-fraud system analyses user behaviour and stops serving ML models for malicious users. It accepts and analyses events coming from mobile apps and the backend.

Defense in depth security measures:

  • We provided security recommendations helping to follow the defence in depth approach for sensitive parts of dataflow, to cover the applicable security standards (like OWASP MASVS 1.3 L2 & R and OWASP ASVS 4.0), and to prioritise security work based on risks&threats profile of the company.

Additional relevant materials

These conference slides explain more business and engineering details about the protecting ML models case. Anastasiia Voitova presented the talk at the OWASP London meetup. The YouTube video is available as well.

Results and outcomes

The resulting IP protection system is multi-layered and runs through applications and gateways of the Customer's systems. It is designed to stay out of sight and not introduce any unnecessary discomforts for developers and end-users. It combines cryptography, mobile application security, API security and anti-fraud modules.

In contrast with many features, security is a context-dependent non-functional requirement that is not something that could be "finally done". Integrating security into the existing system often leads to re-engineering and optimizing some modules, improving them from both security and UX point of view.

As an additional benefit from our engagement, Customer's engineering team had the experience of building sophisticated security controls that run through mobile, backend and cloud.

Up next

Got a challenge that's still standing?

Let’s start a conversation.

Start a conversation

Get whitepaper

Apply for the position

Our team will review your resume and provide feedback
within 5 business days

Thank you!
We’ve received your request and will respond soon.
Your resume has been sent!
Our team will review your resume and provide feedback
within 5 business days