如何通过API抽取数据并存入数据库

通过 axios 调用API

const abcApi = env.CONTEST_API_URL;
const abcApiToken = env.CONTEST_API_TOKEN;

console.log(`${abcApi}  start`);
const { data } = await axios.get(`${abcApi}/` , {
  headers: {
    Accept: 'application/json',
    Authorization: `ApiKey ${abcApiToken}`
  },
  params: {
    resource_id: 93,
    limit: 700,
    order_by: '-end',
  },
});
return data.objects as ContestAPIType[];

其中 ContestAPIType 可以根据返回值定义。

数据加工并保存

 const contests  = await getContest();

const filteredContests = contests.filter((contest) =>
  contest.event.includes("AtCoder Beginner") &&
  contest.n_problems !== null
);

const contestData: Prisma.ContestCreateManyInput[] = filteredContests
  .filter((contest) => contest !== null )
  .map((contest) => ({
    contestExId: contest.id,
    duration: contest.duration,
    start: contest.start,
    end: contest.end,
    event: contest.event,
    host: contest.host,
    href: contest.href,
    n_problems: contest.n_problems,
    n_statistics: contest!.n_statistics,
    abcName: formatAtCoderContestName(contest.href),
  }) as Prisma.ContestCreateManyInput);

await db.contest.createMany({
  data: contestData,
  skipDuplicates: true,
});

通过 filter / map 等方法，对原始数据进行加工，最终保存到数据库，数据库表结构可以和原返回值不一致。

同步循环

function wait(ms: number) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

for( const [index, contest ] of contestData.entries()) {
    console.log(`Processing ${index} of ${contestData.length} records: ${contest.abcName}`);
    const problems = await getProblem(contest.contestExId);

    const problemData: Prisma.ProblemCreateManyInput[] = problems.
      map( (problem) => ({
        problemExId: problem.id,
        n_accepted:  problem.n_accepted,
        n_attempts:  problem.n_attempts,
        n_total:     problem.n_total,
        name:        problem.name,
        rating:      problem.rating,
        short:       problem.short,
        url:         problem.url,
        abcName:     `${problem.short}.${problem.name}`,
        contestExId: contest.contestExId
      }) as Prisma.ProblemCreateManyInput);
    
    await db.problem.createMany({
      data: problemData,
      skipDuplicates: true,
    });

    await db.contest.update({
      where: {
        contestExId: contest.contestExId,
      },
      data: {
        extract_status: 'COMPLETED',
      },
    });

    await wait(10000);
  }

• 用 For of 的方式循环，可以避免异步执行
• 接口有1分钟只能调用10次的限制，自定义一个wait函数等待10秒

---

作者：Bearalise
出处：如何通过API抽取数据并存入数据库
版权：本文版权归作者所有
转载：欢迎转载，但未经作者同意，必须保留此段声明，必须在文章中给出原文链接。