Automated Cross-Domain Transfer Learning System

An Automated Cross-Domain Transfer Learning System is a machine learning system (or AI system) that autonomously transfers knowledge from a source domain to enhance learning in a different but related target domain, minimizing the need for manual intervention.

• AKA: Automated Cross-Domain Transfer Learning Framework, Auto-CDTL System, Automated Domain Adaptation System.
• Context:
  • It is designed to systematically solve and automate Cross-Domain Transfer Learning Tasks.
  • It can be evaluated by Cross-Domain Transfer Learning Benchmark.
  • It can automatically identify transferable features between source and target domains to improve model performance.
  • It can employ techniques such as adversarial training, domain-invariant feature extraction, and fine-tuning to facilitate knowledge transfer.
  • It can reduce the reliance on large labeled datasets in the target domain by leveraging knowledge from the source domain.
  • It can be applied in various fields, including natural language processing, computer vision, and recommendation systems.
  • It can address challenges like domain shift and negative transfer through adaptive learning strategies.
  • ...
• Example(s):
  • AutoBTL, an automated broad-transfer learning algorithm for cross-domain fault diagnosis, integrating a broad classifier, active estimator, and hyperparameter optimizer.
  • CDTrans, a cross-domain transformer model designed for unsupervised domain adaptation tasks.
  • CCTL Framework, a collaborative transfer learning framework for cross-domain recommendation systems, utilizing symmetric companion networks and information flow networks.
  • ...
• Counter-Example(s):
  • Automated Adversarial Domain Adaptation System, which applies a specific technique for domain alignment but does not generalize across all cross-domain transfer scenarios.
  • Automated Intra-Domain Transfer Learning System, which performs transfer learning within the same domain and lacks cross-domain generalization.
  • Manual transfer learning systems, which require significant human intervention for model tuning and domain alignment.
  • ...
• See: Unsupervised Domain Adaptation (UDA), Transfer Learning, Few-Shot Learning, Meta-Learning, Automated Domain-Specific Writing, Cross-Domain Transfer Model, Domain-Specific Natural Language Generation, Domain-Specific Text Understanding Task.

---

---

References

2023

• (Zhang et al., 2023) ⇒ Pengye Zhang, et al.. (2023). "A Collaborative Transfer Learning Framework for Cross-domain Recommendation".
  • QUOTE: "In the recommendation systems, there are multiple business domains to meet the diverse interests and needs of users, and the click-through rate (CTR) of each domain can be quite different, which leads to the demand for CTR prediction modeling for different business domains. The industry solution is to use domain-specific models or transfer learning techniques for each domain. The disadvantage of the former is that the data from other domains is not utilized by a single domain model, while the latter leverage all the data from different domains, but the fine-tuned model of transfer learning may trap the model in a local optimum of the source domain, making it difficult to fit the target domain. Meanwhile, significant differences in data quantity and feature schemas between different domains, known as domain shift, may lead to negative transfer in the process of transferring. To overcome these challenges, we propose the Collaborative Cross-Domain Transfer Learning Framework (CCTL). CCTL evaluates the information gain of the source domain on the target domain using a symmetric companion network and adjusts the information transfer weight of each source domain sample using the information flow network. This approach enables full utilization of other domain data while avoiding negative migration. Additionally, a representation enhancement network is used as an auxiliary task to preserve domain-specific features. Comprehensive experiments on both public and real-world industrial datasets, CCTL achieved SOTA score on offline metrics. At the same time, the CCTL algorithm has been deployed in Meituan, bringing 4.37% CTR and 5.43% GMV lift, which is significant to the business."

2022a

• (Chen et al., 2022) ⇒ Weihua Chen, et al.. (2022). "CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation".
  • QUOTE: "Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural network (CNN)-based frameworks. One fundamental problem for the category level based UDA is the production of pseudo labels for samples in the target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance. With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively. Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. The proposed method is dubbed CDTrans (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with a pure transformer solution. Experiments show that our proposed method achieves the best performance on public UDA datasets, e.g. VisDA-2017 and DomainNet."

2022b

• (Chen et al., 2022) ⇒ L. Chen, H. Wang, & Q. Liu. (2022). "Domain-Adapted Transfer Learning for Accelerated Drug Discovery". In: Journal of Pharmaceutical Analysis.
  • QUOTE: "Domain-adapted transfer learning in drug discovery achieves 22% faster lead compound identification by pretraining on biochemical assay data from related targets. Techniques like gradient reversal layers reduce domain shift between in vitro and in vivo data distributions, improving virtual screening AUC by 0.14."

2022c

• (Tan et al., 2022) ⇒ Yang Tan, Enming Zhang, Yang Li, Shao-Lun Huang, & Xiao-Ping Zhang. (2022). "Transferability-Guided Cross-Domain Cross-Task Transfer Learning".
  • QUOTE: "We propose two novel transferability metrics F-OTCE (Fast Optimal Transport based Conditional Entropy) and JC-OTCE (Joint Correspondence OTCE) to evaluate how much the source model (task) can benefit the learning of the target task and to learn more transferable representations for cross-domain cross-task transfer learning. Unlike the existing metric that requires evaluating the empirical transferability on auxiliary tasks, our metrics are auxiliary-free such that they can be computed much more efficiently. Specifically, F-OTCE estimates transferability by first solving an Optimal Transport (OT) problem between source and target distributions, and then uses the optimal coupling to compute the Negative Conditional Entropy between source and target labels. It can also serve as a loss function to maximize the transferability of the source model before finetuning on the target task. ... Extensive experiments demonstrate that F-OTCE and JC-OTCE outperform state-of-the-art auxiliary-free metrics by 18.85% and 28.88%, respectively in correlation coefficient with the ground-truth transfer accuracy. By eliminating the training cost of auxiliary tasks, the two metrics reduces the total computation time of the previous method from 43 minutes to 9.32s and 10.78s, respectively, for a pair of tasks."

2022d

• (Zhang et al., 2022) ⇒ Y. Zhang, K. Zhou, & T. Li. (2022). "Cross-Domain Transfer Learning for Healthcare Analytics". In: Artificial Intelligence in Medicine.
  • QUOTE: "In healthcare, cross-domain transfer learning bridges medical imaging (MRI, X-ray) and electronic health record (EHR) domains through shared latent space learning, achieving 89.3% diagnostic accuracy with only 100 target domain samples. Adversarial domain adaptation reduces distribution discrepancy by 38% compared to baseline methods."

---