Carrier Routing Network Mining with Improving Timeliness and Accuracy

[
    {
"code":"180",
"desc":"快件到达【xxx营业部】",
"location":{
"city":"xxx市",
"district":"xxx县",
"point":{
"latitude":xxx,
"longitude":xxx
            },
"province":"xxx"
        },
"node":"已揽收",
"opeTitle":"站点装箱",
"time":"2022-09-04 17:29:27"
    },
    {
"code":"xxx",
"desc":"收取快件",
"location":{
"city":"xxx",
"district":"xxx",
"point":{
"latitude":28.65,
"longitude":120.07
            },
"province":"xx"
        },
"node":"已揽收",
"opeTitle":"配送员完成揽收",
"time":"2022-09-04 17:29:27"
    }
]
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
twenty one.
twenty two.
twenty three.
twenty four.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.

1. waybill_no 表示运单号，同一个运单号会有多条节点记录
2. station_index 表示当前这个节点的下标
3. station_enum 表示这个节点的类型，是分拣中心还是揽派网点
4. station_name 表示节点的名称，例如上面例子里的xxx营业部
5. station_status 表示这个节点的状态，例如是进入还是离开
6. operate_time 表示当前节点的操作时间
1.
2.
3.
4.
5.
6.

3.2 Is there really shift information in the track?

The working principle of the carrier network mentions that the carrier will produce by shift. Can you find evidence of shift production from the track results?  Through analysis, we guess that the time for the same flow direction (such as from A sorting center to B sorting center) to leave a certain sorting center (such as leaving A sorting center) should be relatively concentrated.

In real time, through some simple clustering methods, our conjecture is confirmed.  In the figure below, the horizontal axis represents the hour of departure from the sorting center, and each point represents a certain waybill in history. The vertical axis has no business meaning , but is for convenience of display.

The kmeans clustering algorithm is used to draw the above graph, and the kmeans clustering algorithm needs to specify the number of clusters.  Therefore, it is necessary to use algorithms such as Knee/Elbow for cluster number detection, and it is sensitive to outliers, so DBSCAN is finally used in the implementation.  

3.3 How to choose clustering parameters

Although DBSCAN does not need to specify the number of clusters, it needs to specify the distance between points and the density of points. After repeated adjustments, the parameters of these two cores are finally determined as follows:

clustering = DBSCAN(eps=0.25, min_samples=max(5, int(x.size * 0.02)), metric=metric).fit(x_after_reshape)

Where eps is 0.25, which is 15 minutes.  The point density is a maximum of 5 and 2% of the total.

3.4 How to solve the cross-sky problem

From the above cluster diagram, the points of the same wave may appear across days, that is, the time of some points from the distribution center may be 23:50, and the time of some points from the distribution center may be 00:10. The Euclidean distance between these two points is relatively large, so the metrics function of the distance needs to be rewritten.

def metric(x, y):
ret = abs(x[0] - y[0])
if ret > 12:
ret = abs(24 - ret)
return ret
1.
2.
3.
4.
5.

3.5 How the lines are connected in series

It is not enough to analyze the production shifts of the nodes and the shifts of the lines. They also need to be connected in series to obtain the shifts of the finished product lines, so that they can be applied before or during sales. Some simplifications have been made in the processing here. On the one hand, the sorting waves of the sorting center cannot be identified. On the other hand, it is not necessary to pay attention to the sorting waves of the sorting center.

In fact, the process of concatenating finished product line shifts is as follows:

The core code is as follows:

for (int i = 1; i < tmp.getResourceList().size(); ++i) {
    List<NetworkResourceWaveDTO>
next = tmp.getResourceList().get(i)
            .getWaveList();
next.sort(Comparator.comparing(NetworkResourceWaveDTO::getOffTime));
    boolean match = false;
for (NetworkResourceWaveDTO nextWave : next) {
if (nextWave.getOffTime() > p.getEndTime()) {
            match = true;
            duration += nextWave.getDurationDay();
            p = nextWave;
break;
        }
    }
if (!match) {
        duration += next.get(0).getDurationDay() + 1;
        p = next.get(0);
    }
    productLineWave.add(p);
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.

3.6 How is the relationship between the fourth-level addresses and Lanpai outlets established?

From the application point of view, the input condition is the buyer's fourth-level address, but the end point of the carrier network is the delivery site, so it is necessary to establish a mapping relationship between the carrier's delivery site and the fourth-level address. The establishment of the mapping relationship is relatively simple. Among the sites responsible for dispatching the four-level address in the past period of time, the one that dispatches the most orders for this address is selected.

4. The challenge of project landing

Part 3 is more like a theorist's eloquence, so how to implement it in engineering? This includes the development of ODPS SQL, UDF development and DDD, in short, eighteen skills are required.

4.1 How to perform simple machine learning in ODPS

In the process of shift analysis, the clustering algorithm of DBSCAN is used. What if these algorithms are used on odps? In fact, the DBSCAN algorithm has been implemented in python, and odps supports writing UDF in python. It’s just that the current odps operating environment does not install DBSCAN-related packages, so it needs to be installed manually. For the installation tutorial, please refer to the official documentation of Alibaba Cloud.

4.2 Problems of online service

The above cleaning process needs to be run once a day or at least once a week, and the data of a time window in the past is selected for training to obtain the carrier's network, so as to sense changes in the carrier's network in a timely manner. This means that the finished product line, finished product line wave, and node wave information will be regularly updated. In the process of online service, we directly store the data in redis. In order not to occupy too much memory, the memory is optimized by using the hash data structure. Of course, one disadvantage of hash is that the timeout period cannot be set for the field, which means that the data of a certain field of a certain key is actually expired data. , but it will not be deleted, causing a leak, but this leak can be resolved by other technical means.

5. Progress and planning

At present, we have built a third-party carrier network, and the accuracy rate of the first network point prediction is about 65%, and the accuracy rate of the final sorting prediction is about 85%. Future continuous optimization points include: shift  aggregation (for some lines with relatively sparse data, shift aggregation needs to be done), time decay (cleaning data needs to select data from a period of time in the past, and for data that is too old, it should be decayed so that it is in the result The contribution in is smaller), etc., I believe that the accuracy rate can be further improved.