新手学做网站学哪些知识,做微商去哪些社交网站,网站建设方案和报价表,农村自建别墅二层效果图文章目录 FLINK单机版安装部署高于1.9.3需要修改配置文件flink-conf.yaml(低于1.9.3可以跳过)linux启动集群windows下启动Flink实例运行(单机)还有一种方式是上传任务包运行examples\streamingjava: Compilation failed: internal java compiler error高版本启动脚本 FLINK单机… 文章目录 FLINK单机版安装部署高于1.9.3需要修改配置文件flink-conf.yaml(低于1.9.3可以跳过)linux启动集群windows下启动Flink实例运行(单机)还有一种方式是上传任务包运行examples\streamingjava: Compilation failed: internal java compiler error高版本启动脚本 FLINK单机版安装部署
官网下载Downloads | Apache Flink
下载文件历史版本的1.9.3
https://archive.apache.org/dist/flink/flink-1.9.3/flink-1.9.3-bin-scala_2.11.tgz
# 下载Flink安装包笔者这里测试使用Flink1.9版本其它版本下载目录 https://flink.apache.org/downloads.html
wget https://archive.apache.org/dist/flink/flink-1.9.3/flink-1.9.3-bin-scala_2.11.tgz# 解压
tar -xzvf flink-1.9.3-bin-scala_2.11.tgznetCat是linux下自带的支持TCP、UDP、Unix域协议套接字小工具nc命令允许你创建一个连接、侦听另一个连接和传输数据。在windows环境中也可以使用 使用命令nc -lk port号 说明 -l listen监听某个端口 k:保持住当前的连接程序终止的话当前server不断开 功能启动了一个可以发送socket文本流的服务器端口7777
下载地址 netcat 1.11 for Win32/Win64
https://eternallybored.org/misc/netcat/
注意
服务启动依赖jdk环境变量确保已经安装了Java_jdk
加上环境变量
vi source /etc/profile
export JAVA_HOME/opt/zulu_jdk11.68.17
export PATH$JAVA_HOME/bin:$PATHexport PATH/opt/flink-1.9.3/bin:$PATATsource /etc/profile
高于1.9.3需要修改配置文件flink-conf.yaml(低于1.9.3可以跳过)
修改端口号默认端口不影响的可以直接跳过端口配置
进入conf目录下 flink-conf.yaml 修改rest.port 8081端口过于常用容易出现端口冲突笔者这里改为8001
#rest.port: 8081
rest.port: 8001windows下需要加入如下配置,否则执行start-cluster.bat会导致TaskManager进程会退出
taskmanager.cpu.cores: 2
taskmanager.memory.task.heap.size: 512m
taskmanager.memory.managed.size: 512m
taskmanager.memory.network.min: 64m
taskmanager.memory.network.max: 64m加上后会启动JobManager和TaskManager
linux启动集群
进入bin目录执行下面命令
# 启动Flink集群
./start-cluster.sh# 停止Flink集群
#./stop-cluster.sh访问Flink UI链接http://host:8778/#/overview查看dashboard 这里的host为部署flink的机器host
windows下启动
windows当前能支持的最高版本是1.9.3直接安装就行;更高的版本已经不支持windos启动了连启动脚本都已经没有
高版本参考如下仅供参考但是不保证能成功基本会失败
cmd命令行启动
flink-1.9.3\bin.\start-cluster.bat启动后可以在管理页面看到Available Task Slots有一个可用的slots,如果没有参考上面的flink-config.yaml缺少windows下配置项
Flink实例运行(单机)
Flink安装包自带了测试样例这里可以运行WordCount样例程序
flink-\flink-1.9.3\bin ./flink run D:\workspace\IDE\flink-1.9.3\examples\batch\WordCount.jar还有一种方式是上传任务包运行examples\streaming
examples\streaming\SocketWindowWordCount.jar
该样例为从一个ip端口获取数据并进行单词数量统计用来模拟数据流
运行样例分为本地运行命令行和UI界面提交运行也就是分为本地模式和服务器模式两种
先启动端口TCP字符流
nc -lp 8888
./flink sun -c org.apache.flink.streaming.examples.socket.SocketWindowWordCount -p 1 D:\workspace\IDE\flink-1.9.3\examples\streaming\SocketWindowWordCount.jar --host localhost --port 8888
-m:指定主机名后面的端口为 JobManager的 REST 通信端口而不是 RPC的端口RPC通信端口是 6123在提交任务时是通过 REST 端口号(HTTP端口号)将任务上传到 JobManager.
-c: 执行的主类指定 main 方法的全类名
-p:运行的slot实例数最大为当前所有实例数
-jar包路径
-host port :外部参数这里指参数接收的ip和端口号
在管理页面的submit new job中上传 任务jar即可
http://localhost:8001/#/submit
–hostname localhost --port 8888
这里我们依然以example中的WordCount.jar为例提交点击该明细会出现执行的main文件路径合参数配置默认即可点击submit提交
job样例源码
https://gitee.com/jian_yang_lv
flink视频教程
https://www.bilibili.com/video/BV1eg4y1V7AN?p1vd_source439eb7a06dc0a72103551f415ea81556
java: Compilation failed: internal java compiler error
https://blog.csdn.net/weixin_42923363/article/details/126698963
高版本启动脚本
Windows下是没有启动脚本的这里贴上CMD启动脚本
参考资料https://blog.csdn.net/xuexijava85/article/details/114803489
flink.bat
::###############################################################################
:: Licensed to the Apache Software Foundation (ASF) under one
:: or more contributor license agreements. See the NOTICE file
:: distributed with this work for additional information
:: regarding copyright ownership. The ASF licenses this file
:: to you under the Apache License, Version 2.0 (the
:: License); you may not use this file except in compliance
:: with the License. You may obtain a copy of the License at
::
:: http://www.apache.org/licenses/LICENSE-2.0
::
:: Unless required by applicable law or agreed to in writing, software
:: distributed under the License is distributed on an AS IS BASIS,
:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
:: See the License for the specific language governing permissions and
:: limitations under the License.
::###############################################################################echo off
setlocalSET bin%~dp0
SET FLINK_HOME%bin%..
SET FLINK_LIB_DIR%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR%FLINK_HOME%\pluginsSET JVM_ARGS-Xmx512mSET FLINK_JM_CLASSPATH%FLINK_LIB_DIR%\*java %JVM_ARGS% -cp %FLINK_JM_CLASSPATH%; org.apache.flink.client.cli.CliFrontend %*endlocalstart-cluster.bat
这里我把start-cluster.bat放在bin文件夹同级目录中而没有放在bin目录里面
::###############################################################################
:: Licensed to the Apache Software Foundation (ASF) under one
:: or more contributor license agreements. See the NOTICE file
:: distributed with this work for additional information
:: regarding copyright ownership. The ASF licenses this file
:: to you under the Apache License, Version 2.0 (the
:: License); you may not use this file except in compliance
:: with the License. You may obtain a copy of the License at
::
:: http://www.apache.org/licenses/LICENSE-2.0
::
:: Unless required by applicable law or agreed to in writing, software
:: distributed under the License is distributed on an AS IS BASIS,
:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
:: See the License for the specific language governing permissions and
:: limitations under the License.
:: author:zjcjava time:2023/05/24
::###############################################################################echo off
setlocal EnableDelayedExpansion::#####这里我直接使用根目录拼接config和bin目录的path路径
SET FLINK_HOME%cd%
SET bin%FLINK_HOME%\bin
::### SET FLINK_HOME%bin%..
echo ###############################################################################
echo FLINK_HOME %FLINK_HOME%
SET FLINK_LIB_DIR%FLINK_HOME%\lib
SET FLINK_PLUGINS_DIR%FLINK_HOME%\plugins
SET FLINK_CONF_DIR%FLINK_HOME%\conf
SET FLINK_LOG_DIR%FLINK_HOME%\logecho FLINK_CONF_DIR:%FLINK_CONF_DIR%
echo ###############################################################################
SET JVM_ARGS-Xms1024m -Xmx1024mSET FLINK_CLASSPATH%FLINK_LIB_DIR%\*SET logname_jmflink-%username%-jobmanager.log
SET logname_tmflink-%username%-taskmanager.log
SET log_jm%FLINK_LOG_DIR%\%logname_jm%
SET log_tm%FLINK_LOG_DIR%\%logname_tm%
SET outname_jmflink-%username%-jobmanager.out
SET outname_tmflink-%username%-taskmanager.out
SET out_jm%FLINK_LOG_DIR%\%outname_jm%
SET out_tm%FLINK_LOG_DIR%\%outname_tm%SET log_setting_jm-Dlog.file%log_jm% -Dlogback.configurationFilefile:%FLINK_CONF_DIR%/logback.xml -Dlog4j.configurationfile:%FLINK_CONF_DIR%/log4j.properties
SET log_setting_tm-Dlog.file%log_tm% -Dlogback.configurationFilefile:%FLINK_CONF_DIR%/logback.xml -Dlog4j.configurationfile:%FLINK_CONF_DIR%/log4j.properties:: Log rotation (quick and dirty)
CD %FLINK_LOG_DIR%
for /l %%x in (5, -1, 1) do (
SET /A y %%x1
RENAME %logname_jm%.%%x %logname_jm%.!y! 2 nul
RENAME %logname_tm%.%%x %logname_tm%.!y! 2 nul
RENAME %outname_jm%.%%x %outname_jm%.!y! 2 nul
RENAME %outname_tm%.%%x %outname_tm%.!y! 2 nul
)
RENAME %logname_jm% %logname_jm%.0 2 nul
RENAME %logname_tm% %logname_tm%.0 2 nul
RENAME %outname_jm% %outname_jm%.0 2 nul
RENAME %outname_tm% %outname_tm%.0 2 nul
DEL %logname_jm%.6 2 nul
DEL %logname_tm%.6 2 nul
DEL %outname_jm%.6 2 nul
DEL %outname_tm%.6 2 nulfor %%X in (java.exe) do (set FOUND%%~$PATH:X)
if not defined FOUND (echo java.exe was not found in PATH variablegoto :eof
)echo Starting a local cluster with one JobManager process and one TaskManager process.echo You can terminate the processes via CTRL-C in the spawned shell windows.echo Web interface by default on http://localhost:8081/.start java %JVM_ARGS% %log_setting_jm% -cp %FLINK_CLASSPATH%; org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir %FLINK_CONF_DIR% %out_jm% 21
start java %JVM_ARGS% %log_setting_tm% -cp %FLINK_CLASSPATH%; org.apache.flink.runtime.taskexecutor.TaskManagerRunner --configDir %FLINK_CONF_DIR% %out_tm% 21endlocal
