Developing Public Use Longitudinal Synthetic Microdata with Applications to the Survey of Doctorate Recipients

2.1 Initial Disclosure Risk Assessment

Past disclosure risk assessments (Li, Krenzke, and Li 2022) have shown that a CS-PUF alone has low disclosure risk estimates because it contains limited demographic information. In this research, the authors conducted a matching exercise to the population counts in Survey of Earned Doctorates (SED) table products to identify some SDR subgroups that have high ratios of the sample size to the population size and therefore have high reidentification risk. The risk of correct matching may be low for some of the tables or table cells because the variables—for example, citizenship, marital status, and employment—are time sensitive and can change between a respondent taking the SED and taking the SDR. However, the exercise above informed NCSES that the use of external data sources such as the SED would effectively increase the chance of successful reidentification of SDR respondents by data intruders. To measure the file risk associated with combinations of indirect identifiers, the authors used the model-assisted Skinner and Shlomo (2008) approach, which estimates the probability that a sample unique is unique in the population. At the time, the authors measured the longitudinal risk by linking the 2019 and 2021 SDR CS-PUFs through common variables. The Skinner and Shlomo (2008) approach was utilized to measure the reidentification risk while incorporating the longitudinal nature of the data. This approach measured the increase of longitudinal risk relative to the cross-sectional risk. The model-assisted approach that was adapted to quantify reidentification risk in longitudinal survey data is also provided in Li et al. (2021). Next, to assess the relative risk of respondents, on the basis of k-anonymity, the authors investigated the categories of variables that cause a relatively high risk of disclosure by performing thousands of tabulations that resulted in the identification of unique combinations of variables, categories of variables to collapse, variables to suppress, and data values to target if controlled random treatments (e.g., data swapping, perturbation) are considered. Lastly, the continuous variables were reviewed for outliers that may lead one closer to reidentification.

The L-PUF provides longitudinal information, which can increase the disclosure risk in the CS-PUFs, specifically because of the additional information it would make available beyond what is already included in each publicly available CS-PUF. To quantify the potential increase in risk, the records in the L-RUF 2015–19 were matched to the CS-PUFs for the 2015, 2017, and 2019 cycles, respectively. The matching was based on a set of common variables on each file: 5 static demographic variables, 2 static variables related to the first doctorate, and 29 survey variables on each of the three survey cycles (appendix A).

The L-RUF will not be publicly released; however, this initial assessment identified sample uniques in the L-RUF that can be correctly linked to their corresponding unique records in CS-PUFs. They may require a higher level of synthesis and special handling as they pose elevated disclosure risk across files.

This assessment does not directly measure the risk of a data intruder identifying sampled individuals in the population or identifying correctly and uniquely matching cases to those in the sampling frame. The assessment establishes a baseline for the potential increase in disclosure risk and identifies a broader group of sample uniques that could pose heightened risk if the L-RUF were to be released. The above matching approach measures the chance of getting a correct and unique match between L-RUF and CS-PUFs. Once there is a match, then the reidentification risk of those records is the same or similar to that measured in the case of combined information over years across CS-PUFs in Li, Krenzke, and Li (2022).

To quantify the risk of a data intruder identifying correctly and uniquely matching cases to those in the sampling frame of the SDR, matching was performed between the L-RUF 2015–19 to the sampling frame of the 2015 SDR based on seven common indirect identifier variables. The sampling frame is constructed from the SED data, which have never been publicly available. Although the scenario is unlikely, given that the frame is not publicly available, this examination assessed how likely cases can be reidentified in the population.

2.2 Select Synthetic Data

One may synthesize select records of select variables (Reiter and Kinney 2012) to balance the needs of reducing disclosure risk below a tolerable risk threshold and retaining the integrity of the original data (e.g., maintaining the aggregates, distributions, and associations between variables). In this working paper, this approach is denoted as select synthetic data.

Unlike fully synthetic data (Rubin 1993) and partially synthetic data (Little 1993) in which all records are synthesized for all variables or all records are synthesized for some variables, respectively, records with higher disclosure risks were given a higher chance of being synthesized to effectively reduce the risk while limiting the extent of synthetic data generation, reducing the chance that the integrity of the data is compromised. At the same time, lower-risk records are given a lower chance of being synthesized. This results in keeping a proportion of the original data in the file. Note that all records will have a non-zero chance of being selected for synthesis. Select synthetic data can be considered a form of partially synthetic data.

Mechanically, this approach is implemented by creating risk strata and then randomly selecting records within the strata using a predetermined selection rate, which is referred to as the target treatment rate. To ensure comparability across synthesis methods during the research phase, for each target variable, a set of records was randomly selected to be synthesized using the target treatment rates, and the same set of randomly selected records were synthesized across all three synthesis methods described in the Research Phase section.

During the production phase, all variables in the longitudinal file were considered for synthetic data generation, except for those deemed highly analytically important, such as employment status. In contrast, the research phase focused on a selected subset of variables—encompassing both static and longitudinal variables and all types—for synthesis. A comprehensive list of these variables is provided in appendix A, specifying which variables were synthesized in each phase.

2.3 Synthetic Data Generation Approaches under Consideration

As described in the introduction, this research explored three approaches to creating synthetic data. This section describes and provides an overview of the three approaches: the sequential hotdeck approach, the model-assisted constrained hotdeck (MACH) approach, and the use of the R package synthpop.

2.4 Weight Recalibration

The 2015–19 L-RUF underwent weight calibration to align it with known totals, some of which were derived from the sampling frame for the SDR 2015, constructed from the SED data, and some of which were derived from the 2015 cross-sectional restricted use file (CS-RUF) file. Synthetic data generation shifted the weighted distributions for the target synthesized variables. To address this, the weights in the synthetic L-PUF file were calibrated to match the same set of known totals used in weight calibration for the 2015–19 L-RUF. This section details the weight recalibration steps taken after synthetic data generation.

Below is a summary of the process used to create the analysis weights for the 2015–19 synthetic L-PUF file.

1. Base weight: A base weight was the final weight of the original 2015–19 L-RUF and was assigned to every case to account for its selection probability under the two-phase sample design and nonresponse in both the 2017 and 2019 cycles. The final weight of the original 2015–19 L-RUF was derived by calibrating its base weight, which is the product of the 2015 SDR final weight and the reciprocal of the probability of inclusion for the longitudinal sample selection. The replicate weights were created using the successive difference replication method (Fay and Train 1995) and underwent a similar calibration process.
2. Final weight: Raking adjustments used for the original 2015–19 L-RUF were applied to align the weighted sample distribution in the 2015–19 synthetic L-PUF with known distributions. These known distributions include (1) the population distributions of some key frame variables and (2) the weighted sample distributions of selected control variables in the SDR 2015 file, from which the longitudinal sample was drawn. The variables were selected to support the precision level in the LSDR 2015–19 data tables.^{​See the Data Tables for National Center for Science and Engineering Statistics (NCSES). 2022. Survey of Doctorate Recipients, Longitudinal Data: 2015–19. NSF 22-326. Alexandria, VA: U.S. National Science Foundation. Available at https://ncses.nsf.gov/pubs/nsf22326#data-tables.}

The raking variables are shown in table 1. Following this adjustment, the weight distributions were examined for extreme weights that require trimming. Raking adjustments were then performed on the trimmed weights until the weighted totals conformed to the known totals. The replicate weights were raked similarly, but with control totals including random components to account for the variance in the control totals from the cross sectional 2015 SDR file (see Fuller 1998).

2.5 Disclosure Risk Re-Assessment

To assess the risk remaining in the synthetic data, the matching processes described in Section 2.1 (Initial Disclosure Risk Assessment) were repeated on each synthetic data set. It was anticipated that the matching rate would decline because the synthetic data generation process altered some of the matching variables compared with those in the original data set, thereby reducing the likelihood of correct matches.

2.6 Utility

A list of several utility measures were created to assess how well the synthetic data sets reflect the original 2015–19 L-RUF. The utility measures can be broadly grouped into the following categories:

• Quality control (QC) checks: (1) percentage of records changed and (2) check for atypical value combinations across variables
• Weighted frequency checks: (1) distribution of distances from the original values to the synthesized values, (2) high-utility crosstabs, (3) confidence interval overlap, and (4) confidence interval and point estimation
• Measures of associations checks: (1) pairwise associations, (2) significance of regression coefficients, (3) differences between groups, and (4) U-statistic
• Global utility: (1) visual comparisons of top principal components and (2) change in variable importance in principal component analysis

Many of these utility assessment measures are not applied only to the entire data set but also to specific subgroups.

Each of the above named utility measures are described in further detail below.

2.6.2 Weighted Frequency Checks

The first weighted frequency utility measure compared the distance between the original and synthetic values of variables for each individual in the data set. For continuous variables, Euclidean distance was used, and summary statistics of the distances were produced. For categorical variables, Hellinger distance was used. This utility measure checked that the changes for each record and variable were minimal. This measure was applied to each synthesized variable. Part of the checks included comparing the univariate distribution visually through histograms and boxplots.

For combinations of variables considered important, crosstabs were created and the weighted estimate in each cell was compared between the original and synthetic data set. Variable combinations featured in data tables for LSDR 2015–19 (NCSES 2022) were considered important. Since this utility measure can result in many estimates, a plot with the original estimate on the x-axis and the treated estimate on the y-axis was produced to visualize any large deviations. In applying this measure, two sets of variables were used. The first set consisted of derived variables of unemployment spells, employment sector changes, location changes, and working in a field related to the field of doctorate degree. The second set consisted of demographic variables of gender, race and ethnicity, 77-category field of doctorate degree, years since doctorate award, residing location as of 2015, disability status as of 2015, and citizenship status at the time of doctorate award. The derived high-utility crosstabs consisted of three-way crosstabs that included one of the derived variables and two of the demographic variables. The differences between point estimates based on the original and those based on synthetic data relative to the original variance estimates were examined.

For various parameters of importance (e.g., counts, proportions, means) in high-utility crosstabs (LSDR 2015–19 data tables, NCSES 2022), we calculated estimates and confidence intervals using both the original and synthetic data. Ideally, the resulting confidence intervals will mostly overlap so that the same conclusions regarding statistical significance are reached in the synthetic data as in the original data. If the confidence intervals are denoted as (l_o, u_o) and (l_s, u_s) for the original and synthetic data, respectively, then Karr et al. (2006) calculates the confidence interval overlap as follows:

Furthermore, it was determined whether the estimate using the synthetic data is within the confidence interval obtained using the original data, checking that the estimates from the synthetic data are reasonable given the uncertainty of the original data. Specifically, salary was examined for the 3 years within each race and ethnicity category.

2.6.4 Global Utility

Propensity scores were used as a global utility measure for microdata by calculating the U statistic (Woo et al. 2009). To calculate the U statistic, the original and synthetic data are stacked; T = 1 was assigned to the records in the synthetic data, and T = 0 was assigned to records in the original data. A logistic regression model was fitted to T using main effects associated with the synthetic variables. The U statistic was computed as follows:

where n = number of records,  ${\hat{p}}_i$ = propensity score from logistic regression model for record i, and c = proportion of units from the synthetic data file. U should be close to 0 if the original and synthetic files are indistinguishable. The variance estimate of the U statistic was used from Snoke et al. (2018) to enable statistical testing. The U statistic assesses the multivariate associations between variables in the data and examines whether they are similar in both the original data and synthetic data.

2.7 Variance Estimation

In general, to enable inference from select synthetic data, it is important to release multiple implicates of the synthetic data that accounts for the additional variation resulting from the synthesis process. This section describes how to estimate variances and confidence intervals for estimates of parameters of interests, such as the mean or a regression coefficient, when M implicates of synthetic data are released. However, releasing multiple implicates of synthetic data can increase disclosure risk since not all variables across all records are synthesized and data intruders can pinpoint exactly the variables and their records that had values synthesized.

Therefore, to eliminate the increase to disclosure risk, one could release only one implicate of the synthetic data as it is not evident which records are synthesized. However, releasing only one implicate does not allow for variance estimation that accounts for uncertainty in synthesis which may impact inference. In this section, an approach of using a generalized variance function (GVF) was investigated to allow for variance estimation (Wolter 2007) when releasing only one implicate. A GVF is a regression model that describes the relationship between functions of the variance estimates for estimated population parameters of interest as the dependent variable, and potential independent variables such as the estimate itself and the sample size. Given the need to provide a GVF for every target variable that is synthesized, the limitation of this approach is that data intruders will be aware of which target variables are synthesized. However, the data intruders will not know which of the records of the variables have been synthesized. The GVF approach is discussed further in section 2.7.2, with the recommendation and the adaption of a new approach based on replication weights.

2.7.1 Releasing M Implicates

To obtain proper variance estimates and inference when releasing multiple implicates of synthetic data as briefly described above, it helps to follow the work by Burgette and Reiter (2010). By carrying out the synthetic data generation process multiple times (m = 1,2,…,M), M implicates of synthetic data are created. Let Q be the parameter of interest. For example, the population mean of Y or a regression coefficient of Y on X. For each synthetic data set m, Q is estimated by a point estimate $q_m$ and the variance $V$ by $u_m$. According to Burgette and Reiter (2010), the estimate for Q from the M implicates of synthetic data is $\overline{q}$ and the estimate of the variance V is ū + b/M  where the following calculations are needed:

Inference is typically carried out under the t-distribution.

3.1 Initial Risk Assessment

The initial risk assessment revealed that 63% of the L-RUF 2015–19 records were uniquely and correctly matched to their records in all three CS-PUFs even though the L-RUF 2015–19 has a smaller number of survey variables than the cross-sectional files (table 2). It should be noted that the potential for increased disclosure risks due to additional information becoming available was assessed when releasing an L-PUF, given the disclosure risks posed by each publicly available CS-PUF. As mentioned, this assessment does not directly measure the risk of a data intruder identifying sampled individuals in the population or correctly and uniquely identifying matching cases to those in the sampling frame based on SED data. Instead, it establishes a baseline for potential disclosure risk by identifying sample uniques that could pose increased risk if the L-RUF were released. The matching method evaluates the likelihood of correct and unique matches between the L-RUF and CS-PUFs. Once matched, the reidentification risk aligns with that found in previous work on CS-PUFs (Li, Krenzke, and Li 2022). Additional details on the model-assisted approach for estimating reidentification risk in longitudinal data are available in Li et al. (2021). For an example of the risk of disclosure for a publicly available CS-PUF, see Li, Krenzke, and Li (2022).

Using seven common indirect identifier variables—age group, place of birth, gender, race and ethnicity, field of study, region of the PhD award institution, and citizenship status at the time of PhD award—approximately 5% of the 2015–19 L-RUF cases were uniquely and correctly matched to their corresponding records in the sampling frame of the cross sectional 2015 SDR sample. This finding suggests that the reidentification risk is relatively low and not a major concern. Notably, the 63% of the L-RUF 2015–19 records that were uniquely and correctly matched to their records in all three CS-PUFs include these 5% of cases uniquely and correctly matched to their respective records in the sample frame.

3.2 Experimental Factors

The effects of two factors on synthetic data generation—(1) synthetic data generation methods and (2) the degrees of synthesis—were evaluated. The three synthetic data generation methods described in the Methods section were used to assess the impact of the data generating method. Additionally, three sets of target amount of data synthesize (referred as treatment rates) were examined to explore how different degrees of synthetic data generation affect disclosure risk and data utility. As outlined in the Methods section, the selected synthetic data were generated using target treatment rates by risk stratum. These rates are detailed in table 3. For consistent comparison, a set of records was randomly selected and synthesized using the target treatment rates for each scenario, and this same set of records was used across all three synthesis methods. In total, nine scenarios (Sequential hotdeck, MACH, synthpop × 82%, 50%, 15%) were analyzed.

3.3 Select Variables

As shown in appendix A, a subset of variables was selected to focus on during the research phase and the list of target variables was then expanded for the production phase. The research phase variables were chosen based on needing to represent a variety of situations that are present in the full set of variables in the L-RUF. One situation includes whether the variable is static (does not change over time) or longitudinal. Some variables with strong relationships with other variables were selected. For example, some variables are part of skip patterns. Variables of different types were included, such as unordered categorical, ordinal, and continuous.

3.4 Disclosure Risk Re-Assessment

To assess risk in the synthetic data, the same matching process described in the Methods section was repeated on each of the experimental synthetic data sets. As seen in table 4, the percentage of LSDR 2015–19 cases that were uniquely and correctly matched to all three CS-PUFs decreased from 63% (see table 2) to about 5%, 24%, and 50% for treatment rates of 82%, 50%, and 15%, respectively. The extent of synthetic data generation is the main driving factor in how much risk has been reduced, whereas the synthetic data generation approach did not make meaningful differences.

The potential increase in risk is not eliminated, but as mentioned in section 3.1, it should be noted that the risk assessment is based on the match rate to the CS-PUFs, serving as a proxy for the potential for increased risk due to additional information released in the longitudinal file. This does not equate to disclosure risk, as a data intruder would still need to link the record to a population file, which would be a worst-case (or arguably unrealistic) scenario where the intruder would have access to the frame file. Therefore, the risk estimates are overstated. Nonetheless, these measures are useful to help understand the potential reduction in risk.

For the matching to the sampling frame, as in the initial disclosure risk assessment, seven common indirect identifier variables were used for matching: age group, place of birth, gender, race and ethnicity, field of study, region of the PhD award institution, and citizenship status at the time of PhD award. The matching rate was decreased from 5% to about 1%, 3%, and 4% for treatment rates of 82%, 50%, and 15%, respectively.

The risk assessment indicates that the synthetic data generation process has considerably reduced the risks associated with the original data.

3.5 Checks and Utility

Using the utility measures as specified in the Methods section, each of the synthesis methods and treatment rates received a ranking according to each measure. This required each utility measure to be summarized with a single number that summarizes how well the utility measure performed across each of the five implicates for a given treatment. How the rankings were derived for each of the utility measures are described below.

For each of the synthetic continuous variables, each method was ranked by the weighted difference in means between the original and synthetic values, using the weight in the original data (L-RUF). Then, the ranks for each approach were averaged and re-ranked according to the average ranks of the individual variables. The ranks for categorical variables were constructed similarly but using the Hellinger distance instead of the weighted mean.

For the high-utility crosstabs (LSDR 2015–19 data tables), each approach was ranked according to the mean relative absolute difference between the weighted cell total for the original and synthetic data set.

Rankings were created for the confidence interval overlap and indicator for whether the synthetic estimate was located within the original confidence interval at the 95% confidence level according to their respective means. That is, mean of the IO (calculated using the formula given in section 2.6.2) was computed for each variable as well as the percentage of synthesized estimates that were within the original confidence intervals.

The ranking for pairwise associations was determined by taking the mean relative absolute difference of each of the pairwise correlations. For the significance of regression coefficients, the mean consistency of significance of the regression coefficients was considered across implicates for the synthetic data compared with the original data. If a coefficient is statistically significant at the 0.05 significance level (p < 0.05) in both data sets or not significant (p ≥ 0.05) in each data set, it is considered consistent. Lastly, the mean p-value was used across implicates for each treatment for the U statistic (the larger the better).

The 24 options were compared (three treatment rates of 82%, 50%, and 15% crossed with eight synthetic data generation approaches of sequential hotdeck, MACH, and six combinations of synthpop), and table 5 displays the rankings, in which 1 indicates the best option and 24 the worst. The ratios of each utility measure value to the corresponding value for the sequential hotdeck approach at a treatment rate of 15%, which resulted in the best utility overall, are shown in table 6. Both tables illustrate that the treatment rates have a greater impact on utility, compared with the synthetic data generation approaches (e.g., sequential hotdeck vs. MACH). The ranking suggests that the sequential hotdeck approach outperformed the others, followed very closely by MACH, with the combinations of synthpop approaches coming last. The results highlight the potential benefits of leveraging a process tailored for LSDR.

Figure 1 presents a risk-utility plot. The x-axis represents the risk, measured as the percentage of LSDR 2015–19 cases that are uniquely and correctly matched to all three CS-PUFs (table 4), whereas the y-axis represents utility, measured as the average ratio of utility measures to the corresponding measure for the sequential hotdeck approach with an overall synthesis rate of 15% (table 6), across different utility measures. This approach served as the benchmark, as it produced the highest overall utility. To ensure consistency in the interpretation of utility metrics, all measures were transformed so that lower values indicate better utility. For example, metrics indicating closeness (e.g., confidence interval overlap) were converted into measures of distance (e.g., confidence interval non-overlap), and proportions of consistent statistical conclusions were recast as proportions of inconsistent conclusions. The results from the synthpop approach are displayed, specifically from the select synthesis “Synthpop LinReg/CART” option, as it performed the best among all the synthpop combinations.

In figure 1, it is evident that treatment rates have a considerable impact on both risk and utility, more so than the choice of synthetic data generation approach. Although the performance and utility of all three approaches are comparable within a specific treatment rate scenario, the sequential hotdeck and MACH approaches consistently preserved the integrity of the original data slightly better than the synthpop approach.

Given that disclosure risk is defined as the extra information gained by releasing an LSDR data set together with their CS-PUF counterparts, the 50% treatment scenario was found to introduce sufficient uncertainty to mitigate this risk. In terms of data utility, the 50% treatment scenario outperforms the scenario with higher treatment rates. As shown in figure 1, the sequential hotdeck or MACH approach under the 50% treatment scenario falls closest to the bottom-left quadrant of the risk-utility plot—representing lower risk and higher utility—and thus provides the best balance between reducing risk and maintaining data integrity. It should also be factored in that the two approaches consistently demonstrated higher data utility across various evaluation metrics.

Figure 2 compares the weighted percentage of individuals with no changes in employer sector across all three cycles in the original data against the same estimates in two synthetic data sets: one using the MACH approach and the other using the sequential hotdeck approach^{​This type of comparison can be expanded in the final documentation if needed.}; both approaches in the figure used scenario 2, with an overall treatment rate of 50%. The figures display estimates by sex, race and ethnicity, field of degree, and years since doctorate award, as well as the residing location, disability status, and citizenship status as of 2015. Estimates from both synthetic data sets are very close to the original ones with estimates from the MACH approach being slightly closer (average absolute difference of 0.19% vs. 0.21%). This is an example of how well the data integrity was preserved in this particular set of synthesized longitudinal variables, aided by weight recalibration.

3.6 Summary of Research Phase and Selection of Synthetic Data Approach

Uses of synthetic data have been consistently increasing as the demand for access to microdata and privacy concerns grows. For example, synthetic data are seen as a solution for sharing vast amounts of health data toward developing machine learning models and speeding up research on health data while protecting privacy (e.g., Synthea, as detailed in Walonoski et al. 2018). Challenges to generating synthetic data are balancing the reduction of disclosure risk with retaining the integrity of the original data (e.g., maintaining the aggregates, distributions, and associations between variables). To address these challenges, one may synthesize a select set of variables and select records with high disclosure risks, referred to as the “select” synthetic data approach. Three main select synthetic data approaches were applied to the L-RUF for LSDR 2015–19. In this section, a summary is provided of the synthetic data approaches, treatment rates, variance estimation approaches, and decisions made for production based on the findings from the research phase.

Finding 1 (synthetic data generation approach). The MACH approach was selected to produce an L-PUF for LSDR 2015–19.

Each of the three approaches used a hotdeck cell approach to replace original data. The first approach, referred to as the sequential hotdeck approach, is an expansion of the imputation approach tailored to the SDR longitudinal data, taking advantage of existing imputation cells that account for various associations. The second approach, MACH, constrains the distance from the original value and uses predicted mean matching from a main effects model^{​Select interaction effects can be specified.} in the formulation of the hotdeck cells. The third approach includes variations of the R package synthpop, including the CART option which forms hotdeck cells under a model with varying degrees of interaction terms, finding the most important effects on the target variable.

Conceptually, the sequential hotdeck approach can be applied to other data sets; however, it would need to be tailored to each specific data set, especially if the imputation process is not already in place. The MACH approach, on the other hand, can be seamlessly applied to any synthetic data generating application as it is parameter-driven. Although the synthpop approaches use open-source code and can be readily used for other applications to other data sets, it does not give users much control over the modeling process (e.g., users cannot force predictors into the model) and needs tailoring for some options (e.g., parametric regression options require predictors to be selected beforehand).

A thorough and systematic evaluation was conducted to find the best among three approaches. Considering disclosure risk reduction, retention of data utility, and ability to apply to different situations, the best approach was determined to be the MACH approach. The method is developed for synthetic data generation, which conducts a built-in model selection process and synthetic data generation and resulted in the best or the second-best utility in the experiment and similar risk results as other methods. The imputation method performed well for continuous variables mainly because the correlations among salary (or earning) in three different cycles are well-maintained. This was done by systematically respecting longitudinal patterns. For example, to synthesize the salary variable in the 2017 cycle, the salary variable in the 2015 and 2019 cycles were the key variables used in forming imputation cells. This type of tailoring was incorporated into the MACH approach as post-synthesis edits, building on what we have learned from the research phase. Table 7 compares the features of the three synthetic data generation methods, highlighting that the MACH and synthpop method provides capabilities for synthetic data generation such variable selection and modeling.

Finding 2 (treatment rate). The medium treatment rate (overall rate of 50%) was selected to balance risk reduction with retaining data integrity.

Three target treatment rates (proportion of records synthesized) were employed during the research: high (overall rate of 82%), medium (overall rate of 50%), and low (overall rate of 15%). The research results were as expected. They showed that the higher the target treatment rate, the greater the risk reduction (with respect to mitigating the additional information gained by releasing a L-PUF together with cross-sectional counterparts) and the lower the data utility. The authors determined that a 5% reduction in the distance between the original data and synthetic data (increase in utility) across various aspects is worthwhile.

Finding 3 (variable estimation method). To appropriately reflect the variability introduced by synthetic data generation, multiple implicates will be released, rather than the GVF approach that uses only one implicate and correction factors. This was due to the limitation that the GVF approach is too simplistic to allow for more complex modelling. Although the risk reduction due to the synthesizing is not fully realized due to a data intruder knowing which values were synthesized, it allows users to account for the variance component introduced by synthetic data generation in all their analyses.

In general, an estimate of the total variance should account for the synthetic data variance component. Each of the approaches for generating synthetic data has a way of accounting for this term through multiple imputation. The imputation and synthpop approaches repeated the process five times and relied on the random mechanism to infuse the variation between implicates. For the MACH approach, a bootstrap sample of donors was selected within each hotdeck cell. The process was repeated five times.

For the research phase’s recommended approach (MACH), the percentage of the total variance attributable to the synthetic data variance term is 5% to 20% across the three levels of treatment rates. Because the variance term is not negligible, there needs to be a way for users to account for this term in the synthetic public use file (L-PUF). The multiple implicates approach provides a data intruder two pieces of information that some agencies have traditionally suppressed from public knowledge. For one, if the variable has implicates, a data intruder knows which variables were synthesized. Furthermore, if variability exists among the implicates for a given record, then a data intruder knows that those records were synthesized. Therefore, if multiple implicates exist on the file, a data intruder will know the variables and the records that were synthesized. There is some risk in this; however, the intruder will not know the original values. Also, releasing the implicates allows the users to account for the synthetic data error variance component for all types of analyses. The user manual will provide guidelines for using implicates for accurate variance estimation in various statistical programming languages.

4.1 Implementation

The synthetic data generation process is managed through a Master Index File (MIF). The MIF specifies the variables to be synthesized, the partial treatment rates (i.e., the percentage of records to be synthesized) for each risk stratum, and the variables to be included in the pool of candidate predictor variables. It also classifies each variable into one of three types: real numeric, ordered categorical, or unordered categorical. For unordered categorical variables, indicator variables were created. Key features of MACH’s process include:

• Partial treatment rates: These rates were determined based on research phase findings to ensure that approximately 50% of the records were synthesized overall. The partial replacement rates were set at 0.61, 0.41, 0.20, and 0.02 for the four risk strata, from highest risk to lowest risk, as defined in table 2. With the overall rate of 50% and a minimum rate of 2%, the stratum-level replacement rate was determined so that the rates for the second and third strata are approximately one-third and one-sixth of the rate for the riskiest stratum. Although this is somewhat arbitrary, it reflects the structure used to determine the replacement rates at the stratum level.
• Target selection flag: Each variable’s target selection flag was set individually. The flag was set to one if the associated data value was selected for synthesis through a random process. This process involved randomly selecting records based on the partial treatment rate within each risk stratum. Records were sorted by the target variable and key analysis domains, including 5-year award groups for first U.S. science, engineering, or health PhD; gender; race and ethnicity; work status; and residence in the United States, to ensure synthesis occurred across the values of these variables.
• Variable preparation: Predictor variables were recoded as needed for model selection. This step also compiled the pool of predictor variables and created indicator variables for unordered categorical variables.
• Model selection and prediction: After variable preparation, model selection was performed to identify predictors for each target variable and estimate model parameters for generating predicted values. These predicted values were essential for creating hotdeck cells in the synthesis step. The model selection and prediction were done once for each variable using the L-RUF, as the joint distribution among the variables is already given in the original data.
• Hotdeck cells: Hotdeck cells were formed for each target variable by cross-classifying (1) target selection flags indicating records to be replaced, (2) bins created on the target variable to constrain the distance between original and synthesized values, (3) prediction groups based on model predictions (e.g., stepwise linear regression for ordinal and continuous variables, clusters for unordered categorical variables), (4) auxiliary variables accounting for closely related variables, and (5) sampling weight groups. Small cells (cells size less than three) were identified and combined automatically. The five components of the hotdeck cells were sorted in a serpentine manner, and if a cell had fewer respondents than a pre-assigned threshold, it was collapsed with the preceding cell. The rank order of the second to fourth components could be tailored for each target variable via the MIF.
• Synthesis: Target variables were synthesized by drawing from the empirical distribution within each hotdeck cell, without replacement. The approach is model-assisted, utilizing parameters from the model selection process to generate predicted values for forming hotdeck cells.
• Drawing and updating: Within each final hotdeck cell, a with-replacement draw was conducted. Target records were identified by their partial replacement flag, and values were replaced through random draws with replacement from the empirical distribution within the hotdeck cell. Records not targeted for replacement did not contribute values.
• Sequential processing: The predictions and subsequent draws occurred sequentially, with synthetic values used for predictor variables in models for the next variable to be synthesized. Target variables were ordered on the MIF so that those most likely to be predictors were synthesized earlier. The process ran sequentially through all items flagged for synthesis.
• Post-processing checks: After synthesis, pre- and post-synthesis checks were performed to assess the impact, including generating frequencies, means, and correlations before and after synthesis.

The entire process was repeated five times independently with different seeds, resulting in five versions of synthesized values, or implicates.

4.2 Evaluation

This section evaluates the resulting select synthetic data generated using the selected factors: the MACH approach and a synthesis rate of 50% in terms of disclosure risk (defined as the likelihood of data intruders gaining additional accurate information about sampled individuals by matching to the respective cross-sectional PUF) and data utility.

4.3 Variance Estimation

As mentioned, all implicates of the synthetic data will be released to allow for proper variance estimation. Under the replication weights approach, the five implicates are stacked, resulting in a file that has five records for each person. The survey weights aligned with each record are divided by five to retain the total sum of weights. As outlined in appendix B, the variance estimation method is based on replication weights, which were further perturbed to account for variance introduced by synthetic data generation. This method was evaluated by comparing its variance estimates with those derived from the combining rule in Reiter and Kinney (2012). Specifically, two sets of variance estimates were compared for the 64,692‬‬ estimates in the high-utility crosstabs used in the data utility evaluation. The average ratio of the two variance estimates was 1, with values ranging from 0.91 to 1.12, and the distribution was closely centered around this average.