首页
论坛
课程
招聘
[原创]Libc-Database 本地搭建实战
2020-11-7 20:42 3835

[原创]Libc-Database 本地搭建实战

2020-11-7 20:42
3835

0x01 前言

此处,我们以Ubuntu 16.04虚拟机为例讲述搭建方法,为防止环境冲突,请选择使用一台干净的虚拟机,若使用Ubuntu Server 16.04,可能会出现系统安装失败情况,若出现,请参考底部的参考链接。

 

另外,此教程中涉及到了部分文件的修改,我也对原项目进行了部分本地化,因此也可以直接获取我fork修改的项目进行搭建,项目地址:https://github.com/hebtuerror404/libc-database。欢迎Star~

 

本文中最后生成的配置结束的PVM虚拟机已上传至百度云盘,需要的自取即可,链接: https://pan.baidu.com/s/1mfG_ES4GzBrLyoGRerZDoA 密码: 9vpp

0x02 安装环境所需的依赖(需要 ,需要较长⏱️)

1. 更新系统内的所有程序包(需要 ,需要较长⏱️)

使用命令sudo apt-get update && sudo apt-get upgrade -y

 

⚠️:可以使用以下命令换源以加速下载

1
2
3
4
5
sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo vi /etc/apt/sources.list
# vi 内运行以下两条命令
:% s/us.archive.ubuntu.com/mirrors.aliyun.com/g
:wq

2. 通过apt安装必要软件包(需要 )

使用命令

1
2
3
4
5
6
7
sudo apt-get install git vim wget curl python python3 python-pip python3-pip binutils file wget rpm2cpio cpio zstd nginx libffi6 libffi-dev software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
sudo curl https://bootstrap.pypa.io/get-pip.py -o /get-pip.py
sudo python3.6 /get-pip.py
sudo ln -s /usr/local/bin/pip /usr/bin/pip

3. 通过pip安装必要软件包(需要 )

使用命令

1
sudo pip install elasticsearch

4. 安装 docker 以及 docker-compose(需要 )

此处推荐使用以下命令直接获取docker

1
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

docker成功安装后可以去阿里云官网获取镜像加速器,然后按照官网所述直接做如下配置即可。

1
2
3
4
5
6
7
8
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://<镜像加速器ID>.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

然后运行sudo pip install docker-compose -i https://mirrors.aliyun.com/pypi/simple/安装docker-compose

 

⚠️:此处安装完成后,可能产生以下错误

 

image-20200927180508318

 

此时可以使用which docker-compose来确定核心文件位置,然后建立符号链接即可。

5. 安装nodejs以及npm(需要 )

首先运行以下命令获取nodejs

1
2
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs

然后安装npm

1
2
sudo npm install -g n
sudo n latest

image-20200927184204929

0x03 获取 libc-database 项目并处理

1. 获取项目。(需要 )

使用命令git clone https://github.com/niklasb/libc-database.git获取此项目。

2. 修改项目内的Node.js源码。

修改libc-database/searchengine/frontend/src/App.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
import React, { useState, useEffect, useCallback, useRef } from 'react';
import './App.css';
import 'fontsource-roboto';
import Button from '@material-ui/core/Button';
import Grid from '@material-ui/core/Grid';
import TextField from '@material-ui/core/TextField';
import Link from '@material-ui/core/Link';
import CircularProgress from '@material-ui/core/CircularProgress';
import Table from '@material-ui/core/Table';
import TableBody from '@material-ui/core/TableBody';
import TableCell from '@material-ui/core/TableCell';
import TableRow from '@material-ui/core/TableRow';
 
import { makeStyles } from '@material-ui/core/styles';
 
 
- const API_BASE = 'https://libc.rip/api';
+ const API_BASE = 'http://127.0.0.1/api';
 
const api = async (path, data) => {
  let resp = await fetch(`${API_BASE}${path}`, {
    method: 'POST',
    mode: 'cors',
    cache: 'no-cache',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify(data),
  });
  return await resp.json();
};
 
const useStyles = makeStyles((theme) => ({
  root: {
    '& .MuiTextField-root': {
      margin: theme.spacing(1),
    },
    '& .MuiButton-root': {
      margin: theme.spacing(1),
    },
    '& .remove': {
      marginTop: '1.2rem',
      height: '2rem',
    },
    '& .findbutton': {
      marginTop: '1.2rem',
    },
  },
  table: {
    marginTop: '1rem',
    marginBottom: '1rem',
  }
}));
 
 
function SearchRow({ onChange = () => {}, onRemove = () => {} }) {
  const [symbol, setSymbol] = useState("");
  const [address, setAddress] = useState("");
  const [addressValid, setAddressValid] = useState(true);
 
  const onSymbolChange = useCallback((evt) => {
    setSymbol(evt.target.value);
  }, []);
 
  const onAddressChange = useCallback((evt) => {
    setAddress(evt.target.value);
  }, []);
 
  useEffect(() => {
    const valid = !!address.match(/^(0x)?[0-9a-fA-F]*$/);
    setAddressValid(valid);
    onChange({valid, symbol, address});
  }, [address, symbol, onChange]);
 
  return (
    <div>
      <TextField label="Symbol name" value={symbol} onChange={onSymbolChange} />
      <TextField label="Address" error={!addressValid} value={address} onChange={onAddressChange} />
      <Button className="remove" variant="contained" color="secondary" onClick={onRemove}>
        Remove
      </Button>
    </div>
  );
}
 
function SearchForm({ onSearch = () => {} }) {
  const classes = useStyles();
  const [nextId, setNextId] = useState(0);
  const [rows, setRows] = useState([]);
  const [states, setStates] = useState({});
  const onRemoveRef = useRef();
  const onChangeRef = useRef();
 
  const makeRow = (id) => {
    return (
      <SearchRow key={id}
        onRemove={() => onRemoveRef.current(id)}
        onChange={(obj) => onChangeRef.current(id, obj)} />);
  };
 
  const isEmpty = useCallback((i) => {
    let state = states[rows[i].key];
    return !state || (!state.symbol && !state.address);
  }, [rows, states]);
 
  // Add new empty rows automatically
  useEffect(() => {
    let need = true;
    for (let i = 0; i < rows.length; ++i) {
      if (isEmpty(i)) {
        need = false;
        break;
      }
    }
 
    if (need) {
      setRows(rows => rows.concat([makeRow('' + nextId)]));
      setNextId(id => id + 1);
    }
  }, [rows, states, nextId, isEmpty]);
 
  // Remove superfluous rows at the end
  useEffect(() => {
    let i = rows.length - 1;
    while (i >= 1 && isEmpty(i) && isEmpty(i-1)) {
      --i;
    }
    if (i < rows.length - 1) {
      setRows(rows => rows.slice(0, i+1));
    }
  }, [rows, states, nextId, isEmpty]);
 
  const onRemove = useCallback((id) => {
    for (let i = 0; i < rows.length; ++i) {
      if (rows[i].key === id) {
        setRows(rows.slice(0, i).concat(rows.slice(i+1)));
        return;
      }
    }
  }, [rows]);
 
  const onChange = useCallback((id, obj) => {
    setStates({...states, [id]: obj});
  }, [states]);
 
  onChangeRef.current = onChange;
  onRemoveRef.current = onRemove;
 
  const onSubmit = useCallback(() => {
    let symbols = {};
    for (let row of rows) {
      let state = states[row.key];
      if (state && state.valid && state.address && state.symbol) {
        symbols[state.symbol] = state.address;
      }
    }
    onSearch({"symbols": symbols});
  }, [rows, states, onSearch]);
 
  const isValid = useCallback(() => {
    let cnt = 0;
    for (let row of rows) {
      let state = states[row.key];
      if (!state)
        continue;
      if (!state.valid)
        return false;
      if (state.address && state.symbol)
        cnt++;
    }
    return cnt > 0;
  }, [rows, states]);
 
  return (
    <form className={classes.root}>
      {rows}
 
      <div>
        <Button
          disabled={!isValid()}
          variant="contained"
          className="findbutton"
          color="primary"
          onClick={onSubmit}>
          Find
        </Button>
      </div>
    </form>
  );
}
 
function Result({ id, buildid, md5, symbols, download_url }) {
  const classes = useStyles();
  const [open, setOpen] = useState(false);
 
  const onToggle = useCallback((evt) => {
    evt.preventDefault();
    setOpen(!open);
  }, [open]);
 
  let symbolRows = Object.entries(symbols).map(([k, v]) => (
    <TableRow key={k}>
      <TableCell><code>{k}</code></TableCell>
      <TableCell><code>{v}</code></TableCell>
    </TableRow>
  ));
 
  return (
    <div>
      <Link href='#' onClick={onToggle}>{id}</Link>
      {open && (
        <Table size="small" className={classes.table}>
          <TableBody>
            <TableRow>
              <TableCell>Download</TableCell>
              <TableCell>
                <Link href={download_url} download>Click to download</Link>
              </TableCell>
            </TableRow>
            <TableRow>
              <TableCell>BuildID</TableCell>
              <TableCell>{buildid}</TableCell>
            </TableRow>
            <TableRow>
              <TableCell>MD5</TableCell>
              <TableCell>{md5}</TableCell>
            </TableRow>
            {symbolRows}
          </TableBody>
        </Table>
      )}
    </div>
  );
}
 
function App() {
  const [loading, setLoading] = useState(false);
  const [results, setResults] = useState(null);
 
  const onSearch = (data) => {
    setLoading(true);
    (async () => {
      try {
        setResults(await api('/find', data));
      } finally {
        setLoading(false);
      }
    })();
  };
 
  return (
    <div className="App">
      <p>Powered by the <Link href="https://github.com/niklasb/libc-database/tree/master/searchengine">libc-database search API</Link></p>
 
      <Grid container spacing={2}>
        <Grid item sm={12} md={6}>
          <h3>Search</h3>
          <SearchForm onSearch={onSearch} />
        </Grid>
        <Grid item sm={12} md={6}>
          <h3>Results</h3>
          {loading && <CircularProgress />}
          {results !== null && results.map(x => <Result {...x} />)}
        </Grid>
      </Grid>
    </div>
  );
}
 
export default App;

3. 生成此项目。(需要 )

接下来进入项目文件夹并生成Web端主项目

1
2
3
cd libc-database/searchengine/frontend/
sudo npm install
sudo npm run build

⚠️:若npm安装过慢,可以尝试使用cnpm,官网对于cnpm的描述如下:

  1. 这是一个完整 npmjs.org 镜像,你可以用此代替官方版本(只读),同步频率目前为 10分钟 一次以保证尽量与官方服务同步。
  2. registry.npm.taobao.org 安装所有模块. 当安装的时候发现安装的模块还没有同步过来, 淘宝 NPM 会自动在后台进行同步, 并且会让你从官方 NPM registry.npmjs.org 进行安装. 下次你再安装这个模块的时候, 就会直接从 淘宝 NPM 安装了.

要使用cnpm则运行以下命令:

1
2
3
4
npm install -g cnpm --registry=https://registry.npm.taobao.org
cd libc-database/searchengine/frontend/
sudo cnpm install
cnpm run build

0x04 本地获取所有libc(需要 ,需要极长⏱️)

⚠️:若可以在虚拟机内部开启VPN以加速访问境外网站,请在此步开启。

 

使用以下命令获取/更新所有libc文件到本地,此步骤依据网络连接状况需要耗时30 min ~ 2 h,请耐心等待

1
2
3
4
cd libc-database/
./get ubuntu debian centos kali parrotsec
./get arch
./get rpm

./get arch命令可能极其缓慢,属于正常情况,请耐心等待~

 

./get rpm命令可能会因cpio错误而退出,属于正常情况,直接无视即可~

0x05 本地启动查询服务

1. 修改源文件

首先需要修改一系列文件,使之满足启动条件

 

修改libc-database/searchengine/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
 
events {
  worker_connections 768;
}
 
http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
 
  include /etc/nginx/mime.types;
  default_type application/octet-stream;
 
-  ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
-  ssl_prefer_server_ciphers on;
 
  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;
 
  gzip on;
 
  server {
-    listen 443;
+    listen 80;
    server_name libc.rocks;
-    ssl on;
-    ssl_certificate /etc/letsencrypt/live/libc.rip/fullchain.pem;
-    ssl_certificate_key /etc/letsencrypt/live/libc.rip/privkey.pem;
 
    location /api/ {
      proxy_pass http://127.0.0.1:8000;
      proxy_http_version 1.1;
      proxy_set_header   Upgrade $http_upgrade;
      proxy_set_header   Connection keep-alive;
      proxy_set_header   Host $host;
      proxy_cache_bypass $http_upgrade;
    }
 
    location /download/ {
-      alias /home/niklas/libc-database/db/;
+      alias /home/error404/libc-database/db/;
    }
 
    location / {
-      alias /home/niklas/libc-database/searchengine/frontend/build/;
+      alias /home/error404/libc-database/searchengine/frontend/build/;
    }
  }
}

修改libc-database/searchengine/config.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import os
 
ES_INDEX_NAME = 'libcsearch'
 
# ES_HOST = 'localhost'
ES_HOST = 'es01'
 
# DB_DIR = os.path.dirname(os.path.abspath(__file__)) + '/../db'
DB_DIR = '/db'
 
DEFAULT_SYMBOLS = [
    '__libc_start_main_ret',
    'system',
    'dup2',
    'str_bin_sh',
    'read',
    'write',
    'puts',
    'printf',
]
- DOWNLOAD_URL = 'https://libc.rip/download/{}.so'
+ DOWNLOAD_URL = 'https://127.0.0.1/download/{}.so'

修改libc-database/searchengine/api.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
openapi: 3.0.3
info:
  version: 1.0.0
  title: libc-database search engine
  description: ''
paths:
  /find:
    post:
      operationId: app.find
      tags:
      - libcsearch
      description: |-
        Look up libc by various attributes
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Libc'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Libc'
          description: ''
 
  /libc/{id}:
    post:
      operationId: app.dump
      tags:
      - libcsearch
      parameters:
      - in: path
        name: id
        schema:
          type: string
        required: true
      description: |-
        Dump libc symbols
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DumpRequest'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Libc'
          description: ''
 
components:
  schemas:
    Libc:
      type: object
      properties:
        md5:
          type: string
        sha1:
          type: string
        sha256:
          type: string
        buildid:
          type: string
        id:
          type: string
        symbols:
          type: object
          additionalProperties:
            type: string
            pattern: '^(0x)?[a-fA-F0-9]+$'
        download_url:
          type: string
          format: url
          readOnly: true
 
    DumpRequest:
      type: object
      properties:
        symbols:
          type: array
          items:
            type: string
 
 
servers:
-  - url: https://libc.rip/api
+  - url: http://127.0.0.1/api

2. 启动elasticsearch查询服务以及查询后端uwsgi服务(守护服务)

使用命令sudo docker-compose up -d

 

启动后可以使用docker ps检查启动情况

 

image-20200928103911098

3. 建立elasticsearch查询服务索引

/libc-database/searchengine下,执行python3.6 -m index ../db

 

建立完成后,使用curl -XGET 127.0.0.1:9200/_cat/indices/来测试索引建立情况

 

image-20200928142936169

3. 启动本地Nginx服务。

最后,在/libc-database/searchengine下使用/usr/sbin/nginx -c nginx.conf启动本地Ngnix服务

4. 测试服务状态。

使用命令curl -X POST -H 'Content-Type: application/json' --data '{"symbols": {"strncpy": "db0", "strcat": "0x000000000d800"}}' http://127.0.0.1/api/find进行测试。

 

image-20200928143551542

5.Enjoy It

断开虚拟机网络,以验证本地状态下的服务可用性

 

image-20200928144129824

0x06 查询服务的更新(需要 )

执行下列命令即可

1
2
3
4
5
cd libc-database/
./get ubuntu debian centos kali parrotsec
./get arch
./get rpm
python3.6 -m index ../db

0x07 参考链接

【原】光盘安装ubuntu出现BusyBox不能继续安装的终极解决方法


[看雪官方培训] Unicorn Trace还原Ollvm算法!《安卓高级研修班》2021年6月班火热招生!!

收藏
点赞2
打赏
分享
最新回复 (1)
雪    币: 15614
活跃值: 活跃值 (14951)
能力值: (RANK:75 )
在线值:
发帖
回帖
粉丝
Editor 活跃值 2020-11-11 15:03
2
0
感谢分享!期待更多优秀文章!
游客
登录 | 注册 方可回帖
返回